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16.  Abstract 

The  Integrated  Terminal  Weather  System  (ITWS)  will  produce  a  fully  automated, 
integrated  terminal  weather  information  system  to  improve  safety,  efficiency,  and 
capacity  of  terminal  area  aviation  operations.  The  ITWS  will  acquire  data  from  Federal 
Aviation  Administration  (FAA)  and  National  Weather  Service  (NWS)  weather  sensors  as 
well  as  aircraft  in  flight  to  provide  FAA  air  traffic  personnel  with  products  that 
require  no  meteorological  interpretation.  These  products  include  current  terminal 
area  weather  and  near-term  predictions  of  significant  weather  phenomena  (e.g., 
microbursts,  wind  shear,  gust  fronts,  precipitation,  lightning,  terminal  winds,  etc.). 

A  Demonstration/Validation  (DEMVAL)  of  ITWS  prototypes  was  conducted  at  Memphis  and 
Orlando  International  Airports  (MEM) (MCO)  and  the  Memphis  and  Jacksonville  Air  Route 
Traffic  Control  Centers  (ARTCC)  during  the  summer  of  1994.  ITWS  products  were 
evaluated  for  operational  suitability,  usefulness,  and  meteorological  validity. 
Questionnaires  were  completed  by  air  traffic  controllers,  supervisors,  and  traffic 
managers  at  the  completion  of  the  DEMVAL. 

This  report  summarizes  the  results  of  the  meteorological  data  analysis  and 
questionnaire  results,  as  determined  by  ACT-320.  The  resolution  of  Critical 
Operational  Issues  (COI)  are  also  discussed. 

The  results  of  these  analyses  indicated  that  during  the  DEMVAL: 

1.  The  ITWS  products  (a)  were  useful  during  operationally  significant  weather,  (b) 
were  displayed  without  the  need  for  meteorological  interpretation,  and  (c)  enhanced 
supervisor  effectiveness  in  traffic  planning/management  during  adverse  weather; 

2.  Terminal  airspace  and  runways  were  utilized  more  efficiently  when  the  ITWS  was 
operational;  and 

3.  The  ITWS  reduced  perceived  controller  workload  during  adverse  weather  conditions 
in  the  terminal  area. 
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EXECUTIVE  SUMMARY 


The  Integrated  Terminal  Weather  System  (ITWS)  is  a  development 
program  initiated  by  the  Federal  Aviation  Administration  (FAA)  to 
produce  a  fully-automated,  Integrated  terminal  weather 
information  system  to  improve  the  safety,  efficiency,  and 
capacity  of  terminal  area  aviation  operations. 

The  ITWS  will  acquire  data  from  FAA  and  National  Weather  Service 
(NWS)  sensors  as  well  as  from  aircraft  in  flight  in  the  terminal 
area.  The  ITWS  will  provide  products  to  FAA  air  traffic  (AT) 
personnel  that  are  immediately  usable  without  further 
meteorological  interpretation.  These  products  include  current 
terminal  area  weather  and  short-term  (0-30  minutes)  predictions 
of  significant  weather  phenomena. 

The  Operational  Test  and  Evaluation  (OT&E)  that  is  the  subject  of 
this  report,  occurred  during  the  Demonstration/Validation 
(DEMVAL)  Phase  of  the  acquisition  cycle  as  defined  in  FAA  Order 
1810. IF,  and  the  Demonstration  Phase  of  Test  and  Evaluation 
(T&E) .  The  ITWS  program  is  funded  by  the  Aviation  Weather 
Development  Program  (AND-460);  Air  Traffic  (AAT-1)  is  the  program 
sponsor.  The  ITWS  prototype  was  developed  by  Massachusetts 
Institute  of  Technology  Lincoln  Laboratory  (MIT/LL) .  The  DEMVAL 
Operational  Test  and  Evaluation  (OT&E)  was  conducted  by  the  FAA 
Technical  Center,  Weather  Branch,  ACT-320. 

During  the  summer  of  1994,  a  DEMVAL  of  the  ITWS  prototype  was 
conducted  at  Memphis  (MEM)  and  Orlando  (MCO)  International 
Airports.  ITWS  graphics  products  were  displayed  on  Situation 
Displays  (SD)  and  text  products  were  displayed  on  Ribbon  Display 
Terminals  (RBDT)  in  the  tower  and  Terminal  Radar  Approach  Control 
(TRACON)  for  AT  personnel  use  and  evaluation.  SDs  were  also 
installed  at  the  Jacksonville  and  Memphis  Air  Route  Traffic 
Control  Centers  (ARTCC),  NWS  Memphis  Weather  Service  Forecast 
Office  (WSFO)  and  NWS  Melbourne,  FL  Weather  Service  Office,  as 
well  as  Delta  Airlines  (Atlanta)  and  Northwest  Airlines 
(Minneapolis).  The  FAA  Technical  Center  also  had  two  SDs  to 
monitor  the  real-time  ITWS  presentations  for  both  MEM  and  MCO. 

The  ITWS  products  were  evaluated  for  operational  suitability, 
usefulness,  and  meteorological  validity.  The  ITWS  products  were 
evaluated  to  determine  whether  AT  personnel,  without  a 
meteorological  background,  could  effectively  use  each  product  and 
whether  the  products  were  appropriate  for  AT  personnel  use. 

The  Critical  Operational  Issues  (COI)  from  the  ITWS  OT&E  Test 
Plan  for  the  1994  Demonstration  were  also  addressed,  and  based  on 
the  data  collected  from  the  questionnaires,  workload  scales,  and 
interviews,  it  was  determined  that  for  the  current  phase  of 
testing: 
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1.  The  ITWS  products  were  useful  during  operationally 
significant  weather  events, 

2 .  The  ITWS  products  were  displayed  without  the  need  for 
meteorological  interpretation  and  are  easily  understandable, 

3.  The  ITWS  reduced  perceived  controller  workload  during  adverse 
weather  conditions  in  the  terminal  area, 

4.  The  ITWS  products  enhanced  supervisor  effectiveness  in 
traffic  planning/management  during  adverse  weather,  and 

5.  Terminal  airspace  and  runways  were  utilized  more  efficiently 
when  the  ITWS  is  operational. 

Based  on  meteorological  data  collected  and  analyzed  from  the 
DEMVAL,  it  was  determined  that  the  ITWS  products  performed  to  an 
acceptable  level  relative  to  probability  of  detection  (Pp)  and 
false  alarm  rates.  Several  of  the  products  (e.g..  Microburst 
Prediction,  Storm  Cell  Information  attribute  association 
techniques.  Tornado  detection.  Lightning,  and  use  of  the  TDWR  to 
present  the  five  nm  precipitation  range)  require  further 
evaluation  against  the  performance  standards  of  the  ORD. 

Several  of  the  products  neither  enhanced  air  traffic  planning  and 
situational  awareness  nor  hindered  job  performance.  Storm  Cell 
Information  and  Terminal  Weather  Text  Message  were  rated  to  have 
had  no  significant  input  on  situational  awareness  and  planning. 
Responses  to  interviews  and  questionnaires  indicated  that  users 
felt  the  Tornado  and  Lightning  products  were  not  effective  for 
rerouting  air  traffic  since  pilots  automatically  avoid  areas  of 
severe  weather.  However,  these  products  are  useful  for  other 
purposes  (i.e.,  evacuating  the  airport  traffic  control  tower 
(ATCT)  in  the  presence  of  a  tornado,  use  of  the  lightning  product 
for  the  safety  of  ramp  operations  (baggage  handlers,  fuelers, 
etc. ) ) . 

The  results  of  the  DEMVAL  indicate  that  the  ITWS  is  a  valuable 
tool  for  air  traffic  planning  and  management,  but  some  products 
require  improvement.  ACT-320  recommends  that  the  ITWS  program 
proceed  to  Key  Decision  Point  (KDP)-3. 
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1.  INTRODUCTION. 


1.1  PURPOSE  OF  REPORT. 

The  purpose  of  this  Demonstration/Validation  (DEMVAL)  Operational 
Test  and  Evaluation  (OT&E)  report  is  to  discuss  the  conduct  and 
present  the  results  of  the  1994  Integrated  Terminal  Weather 
System  (ITWS)  DEMVAL.  The  DEMVAL  was  conducted  at  Orlando  (MCO) 
and  Memphis  (MEM)  International  Airports  in  the  Airport  Traffic 
Control  Towers  (ATCT)  and  Terminal  Radar  Approach  Control 
(TRACON),  and  at  the  Jacksonville  Air  Route  Traffic  Control 
Center  (ARTCC)(ZJX)  and  Memphis  ARTCC  (ZME).  The  DEMVAL  report 
was  prepared  in  accordance  with  FAA-STD-024b. 

1.2  SCOPE  OF  REPORT. 

This  report  contains  the  test  system  configuration,  the  DEMVAL 
description  and  the  results  (appendix  A)  of  the  DEMVAL  conducted 
at  Memphis  and  Orlando.  It  addresses  the  Critical  Operational 
Issues  (COIs)  described  in  the  ITWS  Operational  Test  and 
Evaluation  (OT&E)  Plan  for  the  1994  Demonstration  Phase 
(hereafter  referred  to  as  1994  DEMVAL  Plan) .  It  also  describes 
how  these  COIs  were  evaluated  during  the  DEMVAL.  Supporting  data 
are  included.  The  report  also  addresses  the  meteorological 
performance  of  the  ITWS  products  and  assesses  the  results  of 
product  performance  (appendix  B) . 

Some  of  the  ITWS  products  presented  during  the  1994 
demonstrations,  such  as  microburst,  gust  front,  hail,  and  tornado 
detection  used  enhanced  or  improved  algorithms  to  present 
products  significantly  different  from  those  previously 
implemented  for  Terminal  Doppler  Weather  Radar  (TDWR)  and  Next 
Generation  Weather  Radar  (NEXRAD).  This  warranted  a  performance 
evaluation  for  each  of  the  enhanced  products .  It  was  beyond  the 
scope  of  the  DEMVAL  to  make  comparisons  between  ITWS  products  and 
existing  products,  such  as  ITWS  Microburst  Detection  versus  TDWR 
Microburst  Detection  or  ITWS  Machine  Intelligent  Gust  Front 
Algorithm  (MIGFA)  versus  TDWR  Gust  Front  algorithm. 

The  FAA  did  not  formally  evaluate  the  ITWS  in  operation  at  the 
National  Weather  Service  (NWS)  and  the  airlines  (Northwest  and 
Delta);  results  of  these  evaluations  are  discussed  in  the  Pre- 
KDP-3  ITWS  Benefits  Quantification,  December  1994. 

1.3  BACKGROUND. 

The  ITWS  program  is  funded  by  the  Aviation  Weather  Development 
Program  Office  (AND-460)  and  is  sponsored  by  Air  Traffic  (AAT-1). 
Previous  ITWS  demonstrations  have  been  conducted  in  1992 
(Orlando)  and  1993  (Orlando  and  Dallas/Fort  Worth) .  The  results 
of  the  1993  demonstration  is  documented  in  the  Final  Report  for 
the  Air  Traffic  Control  (ATC)  Operational  Evaluation  of  the 
Prototype  Integrated  Terminal  Weather  System  (ITWS)  at  Dallas 
Fort/Worth  and  Orlando  Airports  (May- September  1993).  These 


1 


demonstrations  provided  useful  data  that  Massachusetts  Institute 
Of  Technology/Lincoln  Laboratory  (MIT/LL)  used  to  refine  various 
ITWS  algorithms;  they  also  provided  valuable  user  feedback  to  aid 
in  the  conduct  of  the  1994  DEMVAL.  The  products  demonstrated 
were  reguested  by  ATR-400  in  a  memorandum  dated  May  13,  1994. 

In  1993,  the  ITWS  program  commissioned  the  ITWS  Users  Group,  a 
small  working  group  which  included  air  traffic  controllers.  Some 
of  these  controllers  had  worked  with  ITWS  prototypes  and 
recommended  the  design  of  the  human  interface  throughout  the 
development  cycle.  This  group  has  had  significant  input  into  the 
human  interface,  particularly,  the  ITWS  display  design  and 
product  options . 


2 .  REFERENCE  DOCUMENTS 


NAS-SS-1000 

FAA-STD-024b 

FAA  Order  1810. IF 
FAA  Order  1810.4B 


NAS  System  Specification,  Vol  1,  1/94 

Content  and  Format  Requirements  for  the 
Preparation  of  Test  and  Evaluation 
Documentation,  8/94 

Acquisition  Policy,  3/93 

FAA  NAS  Test  and  Evaluation  Policy, 

10/92 

ITWS  Operational  Requirements  Document 
(ORD),  Draft,  9/94 

ITWS  Architecture  Alternatives  Analysis, 
3/94 

ITWS  Test  and  Evaluation  Master  Plan 
(TEMP),  Draft,  9/94 

ITWS  OT&E  Plan  for  the  1994 
Demonstration  Phase,  4/94 

ITWS  OT&E  Procedures  for  the  1994 
Demonstration  Phase 


ATR-400  ITWS  Products  Memorandum,  dated 
May  13,  1994 

Final  Report  for  the  Air  Traffic  Control 
(ATC)  Operational  Evaluation  of  the 
Prototype  Integrated  Terminal  Weather 
System  (ITWS)  at  Dallas/Fort  Worth  and 
Orlando  Airports  (May- Sept ember  1993) 

Description  of  ITWS  Weather  Products, 
MIT/LL,  March  1994 
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Pre-KDP-3  ITWS  Benefits  Quantification, 
Dec.  1994 


3.  SYSTEM  DESCRIPTION. 

3.1  MISSION  REVIEW. 

The  ITWS  is  a  terminal  area  weather  system  which  will  provide  an 
integrated  set  of  safety  and  planning  products  to  air  traffic 
control  (ATC)  personnel  in  towers  and  TRACONs.  Enhancing  flight 
safety,  maintaining  aviation  capacity  during  operationally 
significant  weather,  reducing  weather-related  delays,  and 
decreasing  controller  workloads  will  be  realized  with  the  use  of 
the  ITWS.  Other  users  (ARTCC,  Traffic  Management  Unit  (TMU) , 

NWS,  airlines,  etc.)  should  also  benefit  due  to  increased 
terminal  weather  information  and  common  situational  awareness, 
when  dealing  with  tower/TRACON  ATC  personnel. 

The  ITWS  will  receive  and  process  inputs  from  the  various  FAA  and 
NWS  sensors  and  integrate  them  into  a  single  display  which 
requires  little  or  no  further  meteorological  interpretation  by 
the  user.  ITWS  will  output  its  products  to  the  various  Situation 
Displays  (SD)  and  ribbon  display  terminals  (RBDT)  associated  with 
each  ITWS  host  processor,  which  will  be  housed  in  the  TRACON 
building.  The  ITWS  will  be  deployed  to  34  TRACONs  covering  45 
TDWR  serviced  airports,  Oklahoma  City  (two)  for  training  and 
program  support  facilities,  and  the  FAA  Technical  Center. 

3.2  TEST  SYSTEM  CONFIGURATION. 

The  product  suite  for  the  DEMVAL  (table  3.2-1)  was  comprised  of 
the  ITWS  products  as  defined  in  the  ATR-400  ITWS  products 
memorandum  dated  May  13,  1994.  The  approach  of  the  1994  ITWS 
DEMVAL  was  to  present  products  to  the  user  that  are  planned  to  be 
available  at  Initial  Operational  Capability  (IOC). 

ITWS  SDs  were  located  in  the  tower  and  the  TRACON  at  the  MCO  and 
MEM  airports,  and  in  the  TMU  and  Center  Weather  Service  Unit 
(CWSU)  at  ZME  and  ZJX.  An  SD  was  also  installed  at  Northwest 
Airlines  headquarters  in  Minneapolis,  MN,  and  Delta  Airlines  in 
Atlanta,  GA.  Data  from  Northwest  and  Delta  users  was  collected 
by  MIT/LL,  who  were  involved  in  a  Benefits  Assessment  Study.  Two 
SDs  were  monitored  at  the  FAA  Technical  Center. 

The  ITWS  product  algorithms  that  were  used  during  the  DEMVAL  were 
developed  by  MIT/LL.  These  algorithms  generated  the  products 
listed  in  table  3.2-1  and  described  later  in  this  section.  These 
algorithms  will  be  provided  as  Government  Furnished  Equipment 
(GFE)  to  the  ITWS  production  contractor.  The  software  that 
MIT/LL  utilized  to  run  the  algorithms  and  display  the  products 
will  be  provided  to  the  production  contractor  as  Government 
Furnished  Information  (GFI). 
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The  operating  software  used  during  the  DEMVAL  was  engineering 
level  software;  four  different  programming  languages  were  used. 
The  ITWS  contract  will  specify  that  a  single  higher  order 
computer  language  will  be  used  in  the  production  ITWS  system. 
Since  the  algorithms  will  be  GFE,  product  performance  of  the 
production  ITWS  should  approximate  the  DEMVAL  results .  A  Science 
Panel  was  convened  in  December  1993,  to  evaluate  the  science 
behind  the  ITWS  product  algorithms.  The  algorithms  were  given 
the  panel's  approval.  The  DEMVAL  system  consisted  of  weather 
sensor  and  product  inputs  to  the  MIT/LL  Lexington  office  and 
DEMVAL  field  sites,  where  ITWS  products  were  generated  and 
subsequently  displayed  on  the  ITWS  SD. 

TABLE  3.2-1.  ITWS  DEMVAL  PRODUCT  LIST 


H 

storm  motion  and  extrapolated  position 

2. 

storm  cell  information 

3. 

airport  lightning  with  warning  light 

4. 

microburst  detection  with  countdown  timer 

5. 

microburst  prediction 

6. 

wind  shear  (ITWS  microburst,  ITWS  gust  front) 

7. 

terminal  winds  (Memphis  only) 

8. 

ITWS  precipitation  with  anomalous  propagation  (AP) 
removed 

9. 

ASR-9  precipitation  with  AP  flagged 

10. 

tornado  detection 

11. 

terminal  weather  text  message  (TWTM) 

12. 

long  range  reflectivity  and  long-range  storm  motion 

The  future-state  ITWS  may  include  additional  products  as  required 
by  AT;  it  is  an  adaptable  system,  capable  of  expansion  as 
additional  products  are  developed. 

3.2.1  ITWS  DEMVAL  Product  Description. 

A  brief  description  of  the  ITWS  DEMVAL  products  is  contained  in 
the  following  sections.  Additional  product  information  is 
contained  in  the  MIT/LL  document  "Description  of  ITWS  Weather 
Products . " 

3.2. 1.1  Storm  Motion  and  Extrapolated  Position. 

The  ITWS  storm  motion  product  provides  estimates  of  speed  and 
direction  for  significant  storm  cells  containing  level  three  or 
greater  precipitation.  The  data  source  for  this  product  is  the 
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Airport  Surveillance  Radar  Model  9  (ASR-9)  radar.  Anomalous 
propagation  (AP)  echoes  are  edited  from  the  ASR-9  precipitation 
prior  to  application  of  the  storm  motion  product  (section 
3.2. 1.8).  This  minimizes  the  possibility  of  tracking  AP. 

The  ITWS  storm  extrapolated  position  (SEP)  product  projects 
future  locations  of  the  leading  edge  of  storm  cells  containing 
level  three  or  higher  precipitation.  The  current  leading  edge 
location,  and  the  10-  and  20-minute  projected  leading  edge 
locations  are  displayed. 

The  data  source  for  this  product  is  the  ASR-9  radar.  Input  data 
are  subjected  to  AP  editing  prior  to  SEP  processing,  minimizing 
the  possibility  of  projecting  leading  edges  of  AP. 

3.2. 1.2  Storm  Cell  Information. 

The  Storm  Cell  Information  (SCI)  product  provides  Information  on 
the  presence  of  detected  lightning,  hail,  and  mesocyclones  that 
are  associated  with  individual  storm  cells.  The  SCI  product  also 
provides  echo  top  information  associated  with  storm  cells .  The 
hail  and  mesocyclone  products  were  developed  by  the  National 
Severe  Storms  Laboratory  (NSSL)  in  Norman,  OK,  and  are 
implemented  by  MIT/LL  for  use  with  the  ITWS.  Both  products  are 
intended  as  replacements  for  the  hail  and  mesocyclone  products 
that  are  currently  part  of  the  NEXRAD  product  suite.  When  this 
replacement  occurs,  this  product  will  become  a  "pass  through"  for 
ITWS.  Lightning  was  detected  by  the  National  Lightning  Data 
Network  (NLDN) . 

3. 2. 1.3  Airport  Lightning  With  Warning  Light. 

The  airport  lightning  with  warning  light  product  alerts  air 
traffic  personnel  to  the  presence  of  lightning  in  the  airport 
vicinity.  The  range  is  site  adaptable;  for  the  DEMVAL,  a  10-nm 
range  was  used.  When  lightning  is  present  (as  detected  by  the 
NLDN)  within  10-nm  of  the  airport,  the  lightning  warning  light  is 
illuminated. 

3. 2. 1.4  Microburst  Detection  With  ATIS  Countdown  Timer. 

The  ITWS  microburst  detection  product  provides  locations,  areal 
extent,  and  strength  of  windshear  events.  This  product  also 
provides  runway  oriented  windshear  alert  messages  on  the  RBDT. 

The  data  source  for  this  product  is  the  TDWR.  When  a  windshear 
or  microburst  alert  appears  on  any  RBDT,  the  respective  windshear 
or  microburst  alert  box  is  illuminated  and  indicates  "ACTIVE." 
When  the  alert  is  no  longer  present,  the  countdown  timer  (60 
minutes  for  microburst,  20  minutes  for  windshear)  is  started 
within  the  warning  box. 
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3.2. 1.5  Microburst  Prediction. 


The  microburst  prediction  product  attempts  to  detect  precursors 
to  microbursts  before  the  event  is  detected  by  the  microburst 
detection  product.  A  prediction  made  prior  to  a  detection 
provides  a  longer  lead  time  to  a  microburst  event.  The  data 
sources  for  this  product  are  the  TDWR  and  Meteorological  Data 
Collection  and  Reporting  System  (MDCRS)  soundings. 

3.2. 1.6  Windshear  fITWS  Microburst,  ITWS  Gust  Front). 

The  ITWS  windshear  product  detects  the  presence  of  windshears, 
microbursts,  and  gust  fronts.  It  displays  the  location  and 
strength  of  detected  windshears  and  microbursts  on  the  SD;  it 
provides  location  and  strength  to  the  RBDTs  for  events  detected 
on  active  runways . 

The  gust  front  detection  product  provides  current  locations  and 
estimated  future  detection  of  gust  fronts .  The  product  also 
provides  an  estimate  of  wind  speed  and  direction  behind  the  gust 
front.  The  data  sources  for  this  product  are  the  TDWR  and  Low 
Level  Windshear  Alert  System  (LLWAS). 

3.2. 1.7  Terminal  Winds. 

The  ITWS  terminal  winds  product  provides  wind  speed  and  direction 
at  several  altitudes  and  airport  terminal  corner  posts .  It  is  a 
combination  of  data  from  several  sources:  NEXRAD,  TDWR,  MDCRS, 
LLWAS,  Automated  Surface  Observing  System  ( ASOS ) /Automated 
Weather  Observing  System  (AWOS),  and  Mesoscale  Area  Prediction 
System  (MAPS).  This  product  was  demonstrated  at  MEM  only.  Winds 
at  MCO  during  the  demonstration  time  period  typically  lack 
adequate  variability. 

3. 2. 1.8  ITWS  Precipitation  with  Anomalous  Propagation  (AP) 

Removed . 

The  AP  edited  precipitation  product  is  the  six-level  ASR-9 
precipitation  product  with  AP  removed  (as  determined  by  the 
NEXRAD).  AP  is  a  phenomenon  that  causes  false  echoes  to  appear 
in  the  ASR-9  radar  data.  These  false  echoes  mimic  precipitation 
cells . 

3.2. 1.9  ASR-9  Precipitation  with  AP  Flagged. 

The  ITWS  short-range  precipitation  product  is  the  ASR-9  radar 
weather  channel  six-level  precipitation  with  detected  areas  of  AP 
displayed  in  black  on  the  SD. 

3.2.1.10  Tornado  Detection. 

The  tornado  product  provides  the  location  of  detected  tornadoes. 
An  alert  message  is  also  displayed  on  the  RBDT.  The  data  source 
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for  this  product  is  the  NEXRAD.  The  tornado  product  is  a  pass 
through  product. 

3.2.1.11  Terminal  Weather  Text  Message. 

The  terminal  weather  text  message  product  provides  a  summary 
description  of  terminal  area  weather.  The  text  messages  that  can 
be  selected  for  display  on  the  SD  are  a  textual  representation  of 
the  Terminal  Weather  Information  for  Pilots  (TWIP)  product  that 
can  be  uplinked  to  an  aircraft  cockpit.  The  message  consists  of 
precipitation  intensity,  location,  and  motion  for  cells  within  15 
nm  of  the  airport;  microburst  and  windshear  alerts  with  losses, 
gains,  and  beginning  times;  and  time  and  intensity  of 
precipitation  expected  to  impact  the  area  within  15-nm  of  the 
airport. 

3.2.1.12  Long-Range  Reflectivity  and  Long-Range  Storm  Motion. 

The  ITWS  long-range  precipitation  product  is  the  six-level  NEXRAD 
precipitation.  It  is  considered  to  be  a  "pass  through"  product 
from  the  NEXRAD,  and  as  such,  has  been  previously  tested  and 
evaluated.  The  long-range  storm  motion  product  provides 
estimates  of  motion  (speed  and  direction)  for  level  three  and 
greater  storm  cells,  as  detected  by  the  NEXRAD  for  the  100-  and 
200-nm  ranges. 

3.2.2  Test  System  Changes. 

The  MEM  portion  of  the  DEMVAL  was  delayed  due  to  a  schedule  slip 
in  TDWR  reliability  testing  and  replacement  of  the  TDWR  computer. 
The  TDWR  remained  unavailable  at  the  time  of  the  new  DEMVAL  start 
date.  Since  the  training  of  the  AT  personnel  had  been  completed, 
ATR-400,  AND-460,  and  ACT- 320  agreed  to  start  the  DEMVAL  without 
the  TDWR  available.  (See  appendix  C  for  ITWS  operating  times  in 
Memphis.)  The  last  10  days  of  the  Memphis  DEMVAL  were  also 
conducted  without  an  operational  TDWR,  due  to  TDWR  hardware 
problems  unrelated  to  the  ITWS  DEMVAL.' 

In  addition,  there  were  some  modifications  to  the  ITWS  software 
just  prior  to  the  start  and  during  the  first  week  of  the  DEMVAL. 
Some  of  these  changes  were  made  as  a  result  of  the  decision  to 
start  the  DEMVAL  without  an  operational  TDWR  in  Memphis.  All 
decisions  to  change  products  or  displays  were  made  in  conjunction 
with  AND-460,  ATR-400,  ACT-320,  and  MIT/LL.  These  changes  are 
explained  in  the  following  paragraphs. 

a.  Change  the  response  of  the  display  in  the  absence  of 
windshear  information.  Prior  to  May  20,  1994,  if  the  display 
system  could  not  satisfactorily  acquire  the  windshear  products  it 
showed  a  red  X  across  the  SD  and  indicated  that  the  RBDTs  were 
impaired,  meaning  that  the  ITWS  should  not  be  used.  A  change  was 
required  to  allow  use  of  the  ITWS  without  TDWR  (and  hence, 
without  the  red  X) .  The  red  X  was  removed  in  this  situation  and 
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the  windshear  products  were  added  to  the  blue  "unavailable 
products"  stripe. 

b.  Display  a  clear  indication  that  the  system  is  not  to  be 
used.  When  the  system  was  in  a  warm-up  mode  (just  prior  to  the 
start  of  each  day's  activities),  test  windshear  data  sets  were 
often  passed  through  the  system  to  check  that  software  and 
hardware  were  functioning  correctly.  Products  and  messages  were 
displayed  on  the  SD  and  RBDT  in  this  case.  Likewise,  when  the 
ITWS  was  in  the  process  of  failing  (during  the  course  of  normal 
operations),  a  mechanism  was  required  to  ensure  this  failure 
information  was  displayed. 

In  both  cases,  a  red  X  was  placed  in  the  large  display  window  of 
the  SD  (in  place  of  the  small,  but  not  so  obvious  blue  stripe  at 
the  bottom  of  the  SD) . 

c.  Improve  the  visibility  of  the  Memphis  tower  SD.  AT 
personnel  had  difficulty  viewing  the  SD  in  the  Memphis  tower 
under  certain  ambient  light  conditions.  The  TDWR  monitors  that 
were  being  utilized  lacked  contrast  control.  A  higher  contrast, 
controllable  graphics  display  was  substituted  for  the  TDWR 
monitor  to  correct  the  problem. 

d.  Change  the  RBDT  text  on  the  active  runway  lines  when  no 
windshear  is  available.  Text  was  changed  from  "ITWS  IMPAIRED"  to 
"MBA/WSA  OTS"  (microburst  alert/windshear  alert  out  of  service) 
on  all  active  runway  lines  of  the  RBDT.  The  purpose  was  to 
reflect  the  message  that  no  valid  windshear  products  were 
available,  but  that  the  remainder  of  the  ITWS  system  was 
functional,  including  LLWAS  winds  information. 

e.  Update  the  tornado  alert  line  independent  of  d.  above. 
The  tornado  alert  line  was  changed  to  update  regardless  of  the 
status  of  the  windshear  products.  The  previous  change  made  it 
clear  that  the  ITWS  was  still  functioning  (regardless  of  TDWR 
status)  and  that  tornado  alerts  displayed  on  the  RBDT  were  valid. 

f.  Update  the  status  line  independent  of  d.  above.  This 
line  contains  the  LLWAS  information  (2-minute  running  average  of 
the  centerfield  winds  and  the  gust  report)  and  a  time  stamp. 

LLWAS  continued  to  update  this  information  as  long  as  LLWAS  data 
was  valid.  If  invalid,  the  LLWAS  portion  would  display  "CF  999 
99"  and  report  no  gust  information.  The  time  updated  whenever 
new  data  for  the  runway  lines  or  status  line  was  received.  No 
time  was  reported  if  both  LLWAS  and  TDWR  were  impaired. 

g.  Signal  when  a  complete  failure  of  the  RBDT  occurs.  A 
communications  failure  was  indicated  whenever  all  RBDT 
information  was  lost  (windshear,  tornado,  and  LLWAS).  This  was 
indicated  by  "RBDT  COMM  FAILURE"  displayed  on  all  active  runway 
lines. 
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h.  Change  in  mlcroburst  prediction  algorithm.  When  the 
ITWS  was  presented  to  the  Science  Panel  in  December  1993,  it  was 
recommended  that  further  development  was  required  to  meet  the 
ITWS  functional  requirement  for  a  low  probability  of  false  alarm 
(Pfa) •  Based  on  raw  TDWR  data  from  MCO,  MIT/LL  felt  that  the 
probability  of  detection  (Pd)/  (56  percent)  and  the  PpA  (3 
percent  at  15  kts),  (10  percent  at  30  kts)  were  representative  of 
expected  results.  Therefore,  the  Orlando  DEMVAL  was  conducted 
with  the  microburst  prediction  algorithm  in  "unrestricted  mode." 
Minimal  historic  TDWR  data  was  available  to  evaluate  the 
microburst  prediction  algorithm  performance  for  MEM  prior  to  the 
start  of  the  DEMVAL;  therefore,  the  algorithm  was  run  in 
"restricted  mode."  In  restricted  mode,  the  algorithm  waited  a 
longer  period  of  time  before  declaring  a  microburst  prediction. 
This  served  to  present  more  accurate  predictions,  given  the 
uncertainty  of  algorithm  performance  due  to  the  lack  of  prior 
data.  This  decreased  the  false  alarm  rate  but  also  decreased  the 
amount  of  lead  time,  prior  to  the  actual  microburst. 

i .  Correction  of  reported  echo  tops  in  Storm  Cell 
Information  box.  It  had  been  noted  by  both  the  NWS  Forecast 
Office  in  MEM  and  the  CWSU  at  ZME  that  the  echo  tops  reported  by 
the  Storm  Cell  Information  Product  were  lower  than  those  provided 
by  NEXRAD.  Two  deficiencies  were  found:  (1)  the  ITWS  reported 
echo  tops  were  5000  ft.  low  due  to  a  software  error,  and  (2)  the 
ITWS  echo  tops  for  storms  with  small  areas  of  level  four  or 
greater  precipitation  were  sometimes  low  because  the  maximum 
storm  top  could  be  outside  the  boundary  of  the  highest  level  of 
reflectivity  within  the  cell.  The  echo  top  algorithm  searched 
for  the  highest  cloud  top  within  the  area  of  heaviest 
precipitation.  This  was  corrected  by  increasing  the  minimum 
search  area  for  maximum  echo  top  to  64  km^.  A  detailed 
discussion  of  this  problem  is  found  in  section  5.2.6.10  and  in 
appendix  B. 

3.3  INTERFACES. 

This  DEMVAL  was  not  an  integration  test  of  the  interfaces  between 
the  ITWS  and  other  National  Airspace  Systems  (NAS);  it  was  an 
evaluation  of  the  ITWS  products  and  displays,  and  their  utility 
to  AT  personnel.  However,  to  understand  the  flow  of  ITWS  DEMVAL 
inputs  and  outputs,  the  interfaces  are  discussed  below. 

In  addition  to  the  sensor  interfaces  discussed  below,  other 
inputs  to  the  ITWS  during  the  DEMVAL  Included  satellite  feeds  to 
MIT/LL  Lexington,  MA,  from  the  NLDN  and  Forecast  Systems 
Laboratory  (FSL) .  The  FSL  provided  the  AWOS,  ASOS,  MAPS,  and 
MDCRS  information.  These  DEMVAL  system  interfaces  are  depicted  in 
figures  3.3-1  and  3.3-2.  Figure  3.3-3  shows  the  anticipated  ITWS 
production  data  flow. 
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FIGURE  33-1.  MEMPfflS  ITWS  DEMVAL  CONFIGURATIbN 


NEXRADl  TDWR  I - '  LLWAS  3 
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FIGURE  33-2.  ORLANDO  ITWS  DEMVAL  CONFIGURATION 


System  Inputs  System  User  Outputs 
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FIGURE  3.3-3.  ITWS  PRODUCTION  DATA  FLOW 


Since  this  was  an  ITWS  test  bed  to  evaluate  products,  all 
algorithm  processing  and  product  generation  occurred  at  the 
MIT/LL  Lexington  office  (rather  than  a  TRACON-based  ITWS 
processor) .  Generated  products  were  passed  through  the  field 
sites  at  MEM  and  MCO  prior  to  their  distribution  to  the  users 
listed  in  table  3.3-1. 

TABLE  3.3-1.  ITWS  1994  DEMVAL  SD  LOCATIONS 


Note  1:  Not  operational  ITWS  SDs.  These  were  used  for 
training  purposes  only. 

Note  2:  Non-AT  site.  Real-time  SDs  that 

displayed  the  current  weather  conditions  at  MEM 
and  MCO . 

Note  3:  Operational  SDs  that  were  not  included  in  AT- 

sponsored  DEMVAL. 
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3.3.1  TDWR  Interface. 


Base  (raw  sensor)  data  from  the  TDWR  radars  at  MEM  and  MCO  were 
used  to  generate  all  windshear,  microburst/  and  gust  front 
products.  TDWR  base  data  were  also  inputs  to  the  terminal  winds 
and  terminal  weather  text  message  products . 

3.3.2  NEXRAD  Interface. 

The  NEXRAD  radar  located  in  Melbourne,  FL,  was  used  as  input  to 
the  Orlando  ITWS.  The  NEXRAD  at  Millington,  TN,  provided  input 
data  to  the  Memphis  ITWS.'  NEXRAD  data  were  used  in  the 
generation  of  the  long-range  storm  motion,  storm  cell 
information,  terminal  winds,  AP  edited  precipitation,  long-range 
precipitation,  tornado,  and  terminal  weather  text  message 
products . 

3.3.3  ASR-9  Interface. 

The  ASR-9  radars  located  at  the  MEM  and  MCO  provided  data  used  in 
the  generation  of  the  short-range  precipitation,  storm  motion  and 
extrapolated  position,  storm  cell  information,  and  terminal 
weather  text  message. 

3.3.4  LLWAS  Interface. 

LLWAS  sensors  located  at  the  airports  were  used  to  provide 
centerfleld  and  boundary  threshold  wind  information  on  the  RBDT 
and  to  generate  winds  behind  the  gust  front. 

4.  TEST  AND  EVTUliUATION  DESCRIPTION. 

4.1  TEST  SCHEDULE  AND  LOCATIONS. 

Scheduled  operating  hours  for  the  DEMVAL  were  from  12  noon  until 
7:00  p.m.  local  time,  Monday  through  Friday.  Operations  during 
weekends  and  other  times  were  coordinated  between  the 
participating  facilities  and  local  MIT/LL  personnel,  as  weather 
conditions  dictated. 

All  ITWS  products  were  monitored  by  MIT/LL  personnel  at  field 
sites  in  Memphis  and  Orlando,  as  well  as  from  the  main  laboratory 
in  Lexington,  MA.  ITWS  products  were  monitored  by  FAA  Technical 
Center  personnel  via  SDs  at  the  Technical  Center  and  visits  to 
FAA  and  MIT/LL  facilities  at  MEM,  MCO,  ZJX  and  ZME. 

4.1.1  Memphis  Schedule. 

The  Memphis  portion  of  the  DEMVAL  took  place  from  May  23  through 
July  15,  1994,  at  the  MEM  ATCT/TRACON  and  the  Memphis  ARTCC 
(ZME).  The  Demonstration  was  scheduled  to  commence  on  May  2, 

1994,  but  was  delayed  as  described  in  section  3.2.2.  The  ITWS 
was  scheduled  to  operate  from  Monday  to  Friday,  noon  to  7:00  p.m. 
local  time;  additional  ITWS  operations  were  to  be  on  as  required 
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basis,  as  dictated  by  the  weather  and  MIT/LL  personnel  support 
availability  outside  these  hours.  Due  to  the  frequency  and 
severity  of  the  weather  in  Memphis  during  the  DEMVAL,  and  the 
useful  of  the  ITWS,  the  ITWS  was  operational  11  of  the  16  weekend 
days  and  numerous  hours  outside  the  prescribed  scheduled  time 
period.  The  ITWS  was  operational  for  all  but  these  5  days  of  the 
8 -week  Memphis  portion  of  the  DEMVAL. 

4.1.2  Orlando  Schedule. 

The  Orlando  portion  of  the  DEMVAL  was  conducted  from  July  11 
through  August  19,  1994,  at  MCO  ATCT/TRACON  and  ZJX.  The  ITWS 
was  operational  every  day  during  the  6  week  Orlando  portion  of 
the  DEMVAL. 

4.2  PARTICIPANTS. 

Participants  in  the  DEMVAL  included: 

a.  AND-460,  Aviation  Weather  Development  Program  Office; 
directed,  funded,  and  managed  all  DEMVAL  efforts.  The  Program 
Manager  prepares  and  approves  Program  Directives  (PD)  and  other 
appropriate  agreements  delineating  organizational  activities, 
resources  and  funding,  and  the  DEMVAL  Plan  and  Procedures. 

b.  ATR-400,  Air  Traffic  Requirements,  Program  Sponsor's 
Representative;  set  AT  operational  requirements.  Coordinated 
participation  of  AT  personnel  during  the  DEMVAL,  including 
arrangements  for  those  personnel  administering  the  evaluations  at 
the  FAA  Technical  Center.  Reviewed  and  coordinated  National  Air 
Traffic  Controllers  Association  (NATCA)  approval  of 
questionnaires.  Participated  in  product  evaluation  and  gave 
final  approval  for  the  operational  use  of  products. 

c.  AND-420,  TDWR  Program  Office;  coordinated  maintenance  of 
the  TDWR  radars  with  the  Airway  Facilities  Sector  Field  Office 
(AFSFO),  for  the  noncommissioned  systems. 

d.  ACT-320,  Associate  Program  Manager  for  Test  (APMT); 
responsible  for  the  overall  conduct  of  the  DEMVAL.  The  test  team, 
under  the  direction  of  the  test  director,  was  responsible  for 
data  collection,  monitoring  the  DEMVAL,  and  reporting  results. 
Provided  meteorologists,  AT,  and  human  factors  personnel  for  on¬ 
site  observations  of  ITWS  products  in  operational  setting; 
developed  and  administered  post-test  questionnaires.  Responsible 
for  configuration  control  over  the  system  architecture  and  ITWS 
products.  Reviewed  all  changes  to  products  prior  to  their 
release  for  field  evaluation.  Developed  the  DEMVAL  Plan  and 
Procedures . 

e.  ASD-140,  responsible  for  coordinating  all  Test  NAS 
Change  Proposals  (NCP)  to  support  the  DEMVAL. 
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f.  MEM  ATCT  AT  personnel  evaluated  the  ITWS  products  and 
displays . 

g.  Memphis  ARTCC  AT  personnel  evaluated  the  ITWS  products 
and  displays. 

h.  MCO  ATCT  AT  personnel  evaluated  the  ITWS  products  and 
displays . 


i .  Z JX  AT  personnel  evaluated  the  ITWS  products  and 
displays . 

j.  ATQ-3,  Office  of  Independent  Test  and  Evaluation 
Oversight,  responsible  for  independent  oversight  of  major 
acguisition  programs,  providing  an  independent  assessment  of 
system  performance  and  suitability. 

k.  MIT/LL  supplied  the  ITWS  prototype  and  operating 
hardware  and  software,  maintained  the  equipment,  supplied 
intermediate  products  and  base  data  to  ACT-320  for  product 
evaluation;  operated  Memphis  and  Orlando  field  sites;  and 
provided  safety  monitoring  of  the  ITWS  products . 

4.3  TEST  AND  SPECIALIZED  EQUIPMENT. 

The  equipment  used  during  the  DEMVAL  included  the  RBDTs  and  TDWR 
displays  in  the  tower  and  TRACON.  ITWS  SDs  (supplied  by  MIT/LL) 
were  used  at  the  ARTCCs.  The  TDWR  SD  located  in  the  Memphis 
tower  was  replaced  by  an  ITWS  SD,  which  provided  additional 
contrast  control  required  under  certain  lighting  conditions. 

MIT/LL  coordinated  the  installation  of  all  data  lines  and  input 
sensor  feeds.  Additional  SDs  were  maintained  by  MIT/LL  at  the 
field  sites  and  Lexington  in  order  to  provide  safety  monitoring 
of  ITWS  products  and  to  provide  base  (raw  sensor  data)  and 
intermediate  product  outputs  for  product  performance  analysis. 

The  FAA  Technical  Center  monitored  ITWS  output  to  the  users  via 
SDs  at  the  Technical  Center.  MIT/LL  operated  temporary  field 
sites  at  both  Memphis  and  Orlando,  which  served  as  data 
collection  sites,  where  personnel  observed  and  recorded  ITWS  and 
input  sensor  base  data. 

4.4  TEST  OBJECTIVES /COMPLETION  CRITERIA. 

The  objective  of  the  DEMVAL  was  to  evaluate  ITWS  products  in 
operational  settings.  Of  particular  interest  were  user  interface 
and  ITWS  product  utility,  specifically,  the  interaction  of  AT 
personnel  with  the  ITWS,  how  they  utilized  the  products  to 
perform  their  job  tasks,  the  ease  with  which  they  used  the  ITWS 
SD,  and  how  they  understood  the  displayed  products,  as  well  as 
whether  the  products  were  operationally  useful.  Operationally 
useful  is  defined  as  whether  the  ITWS  products  enhanced  the 
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capability  of  AT  personnel  to  perform  their  jobs.  User  data  were 
collected  via  questionnaires  and  interviews  by  ACT-320. 

COIs  were  developed  and  used  to  evaluate  the  ITWS.  The  following 
COIs  are  stated  in  the  DEMVAL  Plan  and  were  the  focus  of  the  ITWS 
evaluation: 

a.  Are  the  ITWS  products  useful  during  operationally 
significant  weather  in  terms  of  their  availability,  timeliness, 
and  suitability  for  air  traffic  use?  Are  detections  and  false 
alarm  rates  acceptable  to  users? 

b.  Are  the  ITWS  products  displayed  without  the  need  for 
further  meteorological  interpretation?  Is  the  displayed 
information  understandable? 

c.  Does  the  ITWS  reduce  (perceived)  controller  workload 
during  adverse  weather  conditions  in  the  terminal  area? 

d.  Do  ITWS  products  enhance  the  effectiveness  of  traffic 
planning/management  during  adverse  weather  conditions  in  the 
terminal  area?  Are  terminal  airspace  and  runways  planned  for  and 
used  more  effectively? 

e.  Does  the  unavailability  of  interfacing  systems/sensors 
adversely  affect  the  ITWS  operations? 

In  addition,  the  following  completion  criteria  were  to  be 
satisfied,  per  the  DEMVAL  Plan: 

a.  In  Memphis,  10  cases  of  convective  activity  over  4 
different  days,  which  impacted  an  inbound  push; 

b.  Also  in  Memphis,  10  windshear  events  over  4  days,  to 
allow  for  adequate  evaluation  of  microburst  detection,  microburst 
prediction  and  windshear  products  (ITWS  microburst  and  gust 
front) ; 


c.  In  Orlando,  15  cases  of  convective  activity  over  4 
different  days.  (There  are  no  clearly  defined  "push"  times  at 
Orlando  as  there  are  in  Memphis . ) ; 

d.  Also  in  Orlando,  15  windshear  events  over  4  days  for 
adequate  evaluation  of  microburst  and  windshear  products;  and 

e.  Approximately  75  percent  usage  of  the  ITWS  by  qualified 
AT  personnel  during  a  significant  weather  event. 

The  COIs  and  the  completion  criteria  were  developed  by  ACT-320  in 
conjunction  with  AND-460,  ATR-330,  ASD-140,  and  MIT/LL. 

Historical  meteorological  data  indicates  that  MEM  typically 
experiences  15.5  thunderstorms  during  the  months  of  June  and 
July;  Orlando  typically  experiences  9.5  thunderstorms  during  July 
and  August. 
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As  stated  in  the  DEMVAL  Plan,  a  maximum  mean  score  of  2.75  (on  a 
scale  of  1-5,  1  indicating  the  product  consistently  enhances  the 
user's  job  tasks)  on  the  questionnaire  rating  scale  was  required 
to  consider  a  product  acceptable.  Additionally,  the  same 
criteria  were  set  to  determine  the  acceptability  of  ITWS 
functionality. 

Workload  scale  data  were  collected  to  compare  perceived  user 
workload  without  the  ITWS  versus  workload  while  utilizing  the 
ITWS.  Success  criteria  was  an  outcome  that  demonstrated  a 
statistically  significant  decrease  in  perceived  operator  workload 
while  utilizing  the  ITWS. 

In  terms  of  the  first  COI,  meteorological  performance  was 
expected  to  meet  the  requirements  in  the  ITWS  Operational 
Requirements  Document. 

4.5  TESTING  DESCRIPTIONS. 

MIT/LL  personnel  performed  all  hardware  setup  prior  to  the  start 
of  the  DEMVAL,  and  maintained  the  equipment  throughout.  The 
DEMVAL  was  conducted  in  accordance  with  the  DEMVAL  Plan  and 
Procedures.  For  samples  of  the  questionnaire  administered  to  AT, 
see  appendices  C-D  of  the  ITWS  OT&E  Procedures  for  the  1994 
Demonstration . 

AT  personnel  were  observed  using  the  ITWS  in  the  course  of 
performing  their  daily  job  tasks.  Informal  discussions  during 
the  DEMVAL  period  and  structured  interviews  and  questionnaires 
after  the  formal  DEMVAL  ended  were  used  to  formulate  user 
opinions  of  ITWS  product  utility,  availability,  accuracy, 
timeliness,  ease  of  access,  and  interpretation. 

Meteorological  data  were  collected  and  analyzed  for  accuracy  by 
comparing  base  (raw  sensor)  data  and  intermediate  products  with 
ITWS  displayed  products,  as  outlined  in  the  DEMVAL  Plan  and 
Procedures.  MIT/LL  recorded  all  data  received  and  provided  this 
data  to  an  ACT-320  meteorologist  as  requested.  These  data  sets 
were  then  used  to  provide  an  independent  evaluation  of  product 
performance.  Additionally,  meteorological  products  were 
monitored  by  ACT-320  at  the  FAA  Technical  Center  on  real-time  SDs 
in  order  to  evaluate  overall  display  performance  and 
characteristics.  Product  monitoring  and  evaluation  was  also 
conducted  by  ACT-320  at  the  MIT/LL  field  sites. 

4.6  DATA  COLLECTION  AND  ANALYSIS  METHODS. 

Data  were  collected  in  order  to  evaluate  the  ITWS  products  and 
the  COIs  as  they  relate  to  both  human  factors  and  meteorological 
performance.  In  addition  to  the  specific  data  collection  methods 
discussed  below,  the  following  methods  also  were  employed: 
informal  discussions  between  AT  users  and  ACT-320  personnel 
throughout  the  DEMVAL,  direct  observation/monitoring  of  an  SD 
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displaying  real-time  Memphis  and  Orlando  ITWS  products  at  the  FAA 
Technical  Center,  and  field  site  visits  to  monitor  and  evaluate 
ITWS  products . 

4.6.1  Human  Factors  Data  Collection  and  Analysis. 

The  operational  AT  environment  in  which  the  DEMVAL  occurred 
required  human  factors  data  collection  to  be  unobtrusive  to  air 
traffic  controllers  performing  job  tasks.  Therefore,  several 
methods  of  data  collection  were  utilized: 

a.  Evaluation  Questionnaires, 

b.  Subjective  Workload  Scales,  and 

c.  Structured  Interviews. 

The  data  collected  were  used  to  answer  the  COIs  listed  in  section 
4.4.  Additionally,  the  questionnaires  and  structured  interviews 
were  used  to  obtain  data  on  utility,  effectiveness,  performance, 
and  display  issues.  The  questionnaires  and  interviews  also 
provided  rating  scale  data  on  the  functionality  of  the  ITWS 
products.  Results  from  the  workload  scales  provided  a  measure  of 
the  impact  that  the  ITWS  had  on  reducing  perceived  user  workload. 

Structured  interviews  were  conducted  to  better  understand  rating 
scale  responses  and  enable  users  to  expand  upon  their 
questionnaire  responses.  Further  explanation  of  the  techniques 
employed  is  contained  in  the  following  sections. 

Since  completing  the  questionnaire  and  participating  in  the 
interviews  were  voluntary,  not  all  AT  personnel  that  used  the 
ITWS  during  the  DEMVAL  were  formally  polled.  The  number  of 
respondents  varied  for  each  data  collection  method  used  as  shown 
in  table  4.6. 1-1. 
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TABLE  4.6. 1-1.  PERCENTAGES  OF  ELIGIBLE  RESPONDENTS 
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4. 6. 1.1  Evaluation  Questionnaire. 


The  questionnaire  is  a  subjective  rating  scale  measure  that  was 
administered  to  the  ATC  personnel  at  the  end  of  the  DEMVAL.  It 
was  used  to  assess  product  utility,  and  human/ computer  interface 
issues  related  to  the  ITWS  display  design  (color,  graphics,  text, 
overlays,  etc.)  and  system  functionality  (response  times,  update 
rates,  etc.). 

Questionnaires  were  administered  to  users  in  both  the  Terminal 
area  (TRACON  and  Tower)  at  MEM  and  MCO  and  at  the  ZME  and  ZJX. 

The  Terminal  area  users  consisted  of  Traffic  Management 
Coordinators  (TMC),  Supervisors,  and  Controllers  in  Charge 
(CICs).  ARTCC  users  consisted  of  TMCs  and  Supervisors.  The 
questionnaires  for  both  facilities  (ARTCC  versus  Terminal  area) 
were  the  same,  except  for  the  omission  of  the  Tornado  product  and 
Microburst  and  Windshear  Air  Traffic  Information  System  (ATIS) 
countdown  timers  in  the  ARTCC  questionnaire.  Figure  4. 6. 1.1-1  is 
the  five-point  scale  that  was  used  to  rate  the  ITWS  products  as 
displayed  on  the  SD. 


TITLE 

TITLE  DESCRIPTION 

RATING 

CONSISTENTLY  ENHANCES 

CONSISTENTLY  ENHANCES  YOUR  ABILITY  TO  PERFORM 

YOUR  JOB  TASK  WHEN  UTILIZING  THE  PRODUCT;  LIKELY 

TO  LEAD  TO  ENHANCED  JOB  PERFORMANCE. 

1 

FREQUENTLY  ENHANCES 

FREQUENTLY  ENHANCES  YOUR  ABILITY  TO  PERFORM 

YOUR  JOB  TASK  WHEN  UTILIZING  THE  PRODUCT;  MAY 

LEAD  TO  ENHANCED  JOB  PERFORMANCE. 

2 

NEUTRAL 

ENABLES  YOU  TO  DO  YOUR  JOB  TASKS  WHEN  UTILIZING 

THE  PRODUCT;  DOES  NOT  LEAD  TO  DEGRADATION  NOR 

ENHANCEMENT  OF  JOB  PERFORMANCE. 

3 

FREQUENTLY  HINDERS 

FREQUENTLY  HINDERS  YOU  TO  DO  YOUR  JOB  TASK 

WHEN  UTILIZING  THE  PRODUCT;  MAY  LEAD  TO 

DEGRADATION  OF  JOB  PERFORMANCE. 

4 

CONSISTENTLY  HINDERS 

CONSISTENTLY  HINDERS  YOUR  ABILITY  TO  DO  YOUR 

JOB  TASKS;  LIKELY  TO  LEAD  TO  DEGRADATION  OF  JOB 

PERFORMANCE. 

5 

NA 

YOU  HAVE  NEVER  USED  THE  PRODUCT  IN  QUESTION. 

- 

FIGURE  4. 6. 1.1-1.  EVALUATION  QUESTIONNAIRE  RATING  SCALE 
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Due  to  the  number  of  Supervisors  and  TMCs  that  completed  the 
questionnaire,  the  responses  from  these  positions  were  combined 
in  the  interest  of  meaningful  data  analysis,  even  though 
controller  positions  were  identified  in  the  questionnaires  from 
the  Tower/TRACON  facilities.  This  methodology  is  consistent  with 
all  data  analysis  for  each  data  collection  method  used. 

4. 6. 1.2  Workload  Scales. 

Reduction  in  workload  is  one  critical  operational  component  of 
the  ITWS  to  be  investigated.  The  original  intention  of  ACT- 320 
was  to  address  this  COI  by  means  of  direct  observation  of  AT 
personnel  performing  specific  job  tasks.  Data  were  to  be 
collected  prior  to  the  ITWS  installation  and  compared  to  similar 
data  collected  towards  the  end  of  the  DEMVAL.  However,  after 
collecting  this  information  in  MEM  prior  to  the  start  of  the 
DEMVAL,  it  was  determined  that  there  was  no  logical  way  to 
compare  these  data. 

Consequently,  it  was  decided  to  administer  the  Modified  Cooper- 
Harper  (MCH)  Workload  Scale  to  measure  the  impact  of  the  ITWS  on 
perceived  controller  workload.  The  MCH  Workload  Scale  was 
administered  before  and  after  the  OT&E  DEMVAL  in  Orlando.  The 
MCH  was  also  administered  in  Memphis,  although  the  Memphis  DEMVAL 
had  already  started.  Memphis  users  were  asked  to  rate  tasks  and 
workload  based  on  how  they  felt  they  were  affected  prior  to  the 
start  of  the  DEMVAL  (without  the  ITWS)  and  during  the  DEMVAL, 
with  the  ITWS.  Comparison  of  the  results  from  Memphis  did  not 
vary  significantly  from  those  received  in  Orlando. 

This  scale  measures  perceived  workload  using  an  ordinal  rating 
scale  from  1-10  denoting  gradients  of  user  perceived  difficulty 
and  demand  level  (1  =  easy  to  10  =  impossible)  while  performing 
specific  tasks.  Figure  4.6. 1.2-1  depicts  an  example  of  the  MCH 
Scale. 
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DIFFICULTY  LEVEL  OPERATOR  DEMAND  LEVEL 


VERY  EASY.  OPERATOR  MENTAL  EFFORT  IS  MINIMAL  AND 

HIGHLY  DESIRABLE  DESIRED  PERFORMANCE  IS  EASILY  ATTAINABLE 

1 

EASY,  OPERATOR  MENTAL  EFFORT  IS  LOW  AND 

DESIRABLE  DESIRED  PERFORMANCE  IS  ATTAINABLE 

2 

FAIR.  ACCEPTABLE  OPERATOR  MENTAL  EFFORT  IS  REQUIRED 

MILD  DIFFICULTY  TO  ATTAIN  ADEQUATE  SYSTEM  PERFORMANCE 

3 

MINOR  BUT  ANNOYING  MODERATELY  HIGH  OPERATOR  MENTAL  EFFORT  IS 

DIFFICULTY  REQUIRED  TO  ATTAIN  ADEQUATE  SYSTEM  PERFORMANCE 

4 

MODERATELY  OBJECTIONABLE  HIGH  OPERATOR  MENTAL  EFFORT  IS  REQUIRED 

DIFFICULTY  TO  ATTAIN  ADEQUATE  SYSTEM  PERFORMANCE 

5 

VERY  OBJECTIONABLE  BUT  MAXIMUM  OPERATOR  MENTAL  EFFORT  IS  REQUIRED 

TOLERABLE  DIFFICULTY  TO  ATTAIN  ADEQUATE  SYSTEM  PERFORMANCE 

6 

MAJOR  DIFFICULTY  MAXIMUM  OPERATOR  MENTAL  EFFORT  IS  REQUIRED 

TO  BRING  ERRORS  TO  MODERATE  LEVEL 

7 

MAJOR  DIFFICULTY  MAXIMUM  OPERATOR  MENTAL  EFFORT  IS  REQUIRED 

TO  AVOID  LARGE  OR  NUMEROUS  ERRORS 

8 

MAJOR  DIFFICULTY  INTENSE  OPERATOR  MENTAL  EFFORT  IS  REQUIRED 

TO  ACCOMPLISH  TASK.  BUT  FREQUENT  OR 

NUMEROUS  ERRORS  PERSIST 

9 

IMPOSSIBLE 

INSTRUCTED  TASK  CANNOT  BE  ACCOMPLISHED 

10 

RELIABLY 

FIGURE  4.6. 1.2-1.  MODIFIED  COOPER-HARPER  WORKLOAD  RATING  SCALE 
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During  the  administration  of  the  MCH  Workload  Scale,  users  were 
asked  to  rate  their  workload  while  performing  these  tasks  under 
conditions  with  and  without  the  ITWS.  Both  of  these  conditions 
consisted  of  (1)  operationally  significant  weather,  and  (2) 
moderate  to  heavy  AT  flow.  The  six  tasks  that  were  rated 
included: 


a.  Planning  AT  flow 

b.  Planning  airspace  use 

c.  Decisions  on  rerouting  aircraft 

d.  Avoidance  of  adverse  weather 

e .  Changing  runways 

f.  Choosing  runways 

4 . 6 . 1 . 3  Structured  Interviews . 

At  the  end  of  the  DEMVAL,  structured  interviews  were  conducted 
with  the  users.  The  interview  questions  were  designed  to  address 
human  factors  COIs.  These  issues  included:  effects  on  traffic 
management,  planning  and  decision  making,  product  utility, 
meteorological  interpretability,  availability,  effectiveness, 
timeliness,  and  performance. 

There  was  some  modification  to  the  questions  that  were  asked  of 
the  ARTCC  users  to  accommodate  differences  between  the  ARTCC  and 
Tower  environments.  Differences  are  noted  in  the  results 
section. 

4.6.2  Meteorological  Data  Collection  and  Analysis. 

Meteorological  data  used  in  this  report  were  obtained  from  two 
sources.  The  ACT-320  meteorologist  who  visited  the  test  sites  at 
Memphis  and  Orlando  maintained  a  log  of  observed  events  and 
MIT/LL  provided  ACT-320  with  base  data  and  products  in  the  form 
of  color  plots  and  operator  logs  for  post-demonstration  analysis. 

At  the  test  sites,  MIT/LL  maintained  a  series  of  displays  of 
real-time  base  data  and  intermediate  products  in  addition  to  the 
ITWS  SD.  With  these  displays,  for  example,  it  was  possible  for 
the  meteorologist  to  check  the  TDWR  base  velocity  and 
reflectivity  displays  for  microburst,  windshear,  or  gust  front 
signatures  to  corroborate  ITWS  detections  shown  on  the  SD.  This 
information  also  aided  in  the  selection  of  data  for  post¬ 
demonstration  analysis. 

The  meteorologist's  log  contained  entries  related  to  occurrences 
of  meteorological  events  such  as  storm  cells,  microbursts, 
windshears,  gust  fronts,  hail,  and  other  pertinent  weather 
phenomena.  When  possible,  the  meteorologist  noted  the  type  of 
event,  time,  strength,  and  location  relative  to  the  airport 
center  or  runways,  and  whether  an  ITWS  event  was  evident  in  base 
data.  The  meteorologist  also  noted  possible  problems  such  as 
system  failures  and  data  source  and  product  unavailability. 
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In  order  to  facilitate  an  independent  evaluation  of  product 
performance,  ACT-320  requested  that  MIT/LL  provide  data  and 
analysis  tools  in  the  same  form  or  in  a  form  similar  to  those 
they  have  used  in  their  own  evaluations  of  the  products.  Results 
of  the  ACT-320  evaluation  of  the  meteorological  performance  are 
discussed  in  section  5.2.6.  Appendix  B  contains  a  description  of 
the  analyses  performed  for  each  product  evaluation. 

5.  RESULTS  AND  DISCUSSION. 

The  results  of  the  data  collection  and  analysis  are  discussed  in 
this  section.  The  completion  criteria  and  the  questionnaire/ 
interview  and  meteorological  evaluation  results  as  they  relate  to 
the  COIs  are  also  discussed. 

There  were  no  major  problems  with  the  ITWS  during  the  DEMVAL 
periods  at  either  Memphis  or  Orlando.  The  readability  of  the 
tower  SD  in  Memphis  and  the  minor  algorithm  and  display  problems 
that  occurred  were  discussed  in  earlier  sections  (section  3.2.2). 

Appendix  C  contains  a  table  of  the  operating  times  of  the  ITWS 
and  the  major  input  sensors  during  the  Memphis  DEMVAL.  This  table 
shows  that  the  ITWS  was  operational  for  all  but  5  weekend  days  of 
the  DEMVAL  period.  Except  for  the  start  of  the  test  (when  it  was 
known  that  the  TDWR  was  unavailable)  and  the  last  week,  when  the 
TDWR  was  not  operational,  TDWR  availability  was  acceptable.  This 
table  shows  that  overall  availability  of  the  TDWR  during  the  530 
hours  of  ITWS  operations  was  71  percent.  However,  of  the  13  days 
when  TDWR  was  unavailable,  there  were  5  days  when  there  was  no 
significant  weather;  thus  the  availability  of  the  TDWR  during 
significant  weather  exceeded  82  percent.  NEXRAD  availability  was 
96  percent  and  ASR-9  weather  channel  availability  was  100 
percent.  Both  radars  had  excellent  availability. 

Appendix  D  contains  a  table  of  the  operating  times  of  the  ITWS 
and  the  major  input  sensors  in  MCO.  This  table  shows  that  the 
ITWS  was  operational  every  day  of  the  6  week  DEMVAL  period,  for  a 
total  of  297  hours.  The  input  sensor  availability  was  excellent; 
96  percent  for  TDWR,  94  percent  for  NEXRAD,  and  100  percent  for 
ASR-9. 

5.1  COMPLETION  CRITERIA. 

The  DEMVAL  Plan  set  goals  for  minimum  numbers  of  AT  users  and 
minimum  occurrences  of  weather  events  to  assure  that  users  had 
ample  opportunity  to  utilize  the  ITWS  products.  The  completion 
criteria,  as  defined  in  the  DEMVAL  Plan,  for  each  DEMVAL  site 
were  satisfied  as  discussed  in  the  following  sections. 

5.1.1  Memphis  Completion  Criteria. 

The  requirement  stated  in  the  DEMVAL  Plan  was  10  cases  of 
convective  weather  during  inbound  pushes  and  10  windshear  events 
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in  Memphis.  During  periods  of  ITWS  operations  in  Memphis,  there 
were  42  cases  of  convective  activity  during  inbound  pushes;  an 
additional  42  cases  of  convective  activity  occurred  during 
outbound  pushes.  These  events  occurred  over  41  days.  This 
exceeds  the  requirement  for  10  convective  events. 

There  were  numerous  wlndshear  events  in  the  TRACON  area;  a  subset 
of  these  events  contains  the  windshears,  microbursts,  and  gust 
fronts  that  impacted  the  airport.  There  were  45  wind  events  that 
impacted  the  airport:  15  windshears,  4  microbursts,  and  26  gust 
fronts.  These  numbers  (from  the  more  restrictive  subset)  satisfy 
the  minimum  requirement  of  10  windshear  events. 

Another  method  of  calculating  the  Impact  that  weather  had  on  the 
airport  during  the  DEMVAL  is  to  determine  the  number  of  runway 
arenas  affected  by  windshears,  microbursts,  and  gust  fronts.  An 
arena  is  defined  as  a  critical  region  in  the  terminal  area  (e.g., 
final  approach  or  departure  corridor  or  runway).  For  example,  if 
a  microburst  was  present  on  runway  18L,  two  arenas  would  be 
affected;  18L  arrival  and  18L  departure.  If  a  microburst  was  at 
1  mile  final  from  the  arrival  end  of  the  same  runway,  only  one 
arena  would  be  affected  -  18L  arrival.  A  single  microburst  that 
was  situated  over  two  different  runways  would  affect  four  arenas 
-  arrivals  and  departures  for  each  runway. 

In  total,  550  arenas  were  affected  during  the  MEM  portion  of  the 
DEMVAL.  Although  there  were  no  minimum  criteria  defined  for 
runway  arenas  impacted,  it  is  clear  from  this  information  that  a 
sufficient  quantity  of  weather  was  experienced  during  the  Memphis 
DEMVAL  period. 

Seventy-five  percent  of  qualified  Memphis  AT  personnel  were 
required  to  have  used  the  ITWS  during  significant  weather  to 
satisfy  the  completion  criteria.  Greater  than  95  percent  of 
qualified  MEM  AT  personnel  used  the  ITWS  during  adverse  weather; 
this  number  was  based  on  observations  and  discussions  with  AT 
managers . 

5.1.2  Orlando  Completion  Criteria. 

There  were  28  days  during  the  MCO  portion  of  the  DEMVAL  when 
convective  activity  impacted  the  airport  ( 15  cases  of  convective 
activity  was  the  minimum  required) .  There  was  no  requirement  for 
weather  events  to  occur  during  pushes,  since  Orlando  does  not 
typically  experience  defined  pushes  like  MEM;  MCO  departures  and 
arrivals  occur  at  a  more  steady  rate.  (For  purposes  of  this 
report,  a  push  is  defined  as  a  period  of  intense  arrival  or 
departure  activity,  lasting  from  1  hour  to  an  hour  and  a  half.) 

A  total  of  121  wind  events  impacted  the  airport  at  MCO:  58 
windshears,  17  microbursts,  and  46  gust  fronts.  This  satisfies 
the  requirement  of  15  windshears.  A  total  of  1593  runway  arenas 
(as  defined  in  section  5.1.1)  were  impacted  during  the  MCO 
DEMVAL.  Greater  than  95  percent  of  the  qualified  MCO  AT 
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personnel  used  the  ITWS  while  adverse  weather  was  present  during 
the  DEMVAL;  this  number  was  based  on  observations  and  discussions 
with  AT  managers . 

5.2  CRITICAL  OPERATIONAL  ISSUES  (COI^  RESULTS. 

The  COIs  stated  in  section  4.4  were  answered  by  analyzing 
questionnaire  and  workload  data  and  by  analysis  of  the 
meteorological  data  and  ITWS  product  performance.  Meteorological 
performance  of  the  ITWS  products  is  also  discussed.  In  the 
following  sections,  appropriate  COIs  are  in  boldface  type.  The 
meteorological  analysis  performed  with  respect  to  the  first  COI 
is  addressed  separately  in  section  5.2.6.  Additional  information 
from  the  interviews  and  questionnaires  are  included  in  appendix 
A;  additional  meteorological  information  is  contained  in  appendix 
B. 


5.2.1  Usabilitv/Suitabilitv/Timeliness  COI. 

Are  the  ITWS  products  useful  during  operationally  significant 
weather  in  terms  of  their  availability,  timeliness,  and 
suitability  for  AT  use?  Are  detection  and  false  alarm  rates 
acceptable  to  users?  (The  second  question  is  related  to 
meteorological  performance  and  is  discussed  separately  in  section 
5.2.6. ) 

Data  collected  from  the  questionnaires  relative  to  product 
utility  were  used  to  generate  the  mean  rating  scores  found  in 
figure  5. 2. 1-1  and  table  5. 2. 1-1.  Figure  5. 2. 1-1  is  a  bar  graph 
of  the  mean  scores  of  product  utility  for  all  responding  users. 
The  product  utility  is  rated  well  below  the  maximum  mean  (2.75). 
Table  5. 2. 1-1  and  figure  5.2. 1-1  are  based  on  a  five-point  scale, 
where  1  =  product  consistently  enhances  job  performance  and  5  = 
product  consistently  hinders  job  performance  (figure  4. 6. 1.1-1). 

As  shown  in  table  5.2. 1-1,  the  total  mean  rating  for  the  utility 
of  all  products  was  1.76.  Ratings  for  each  individual  product 
were  significantly  lower  than  the  maximum  mean  (2.75)  for  product 
suitability. 
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TABLE  5. 2. 1-1 


MEAN  PRODUCT  UTILITY  RATINGS 


+  =  No  Terminal  Winds  Product  in  Orlando 
++  =  No  ATIS  Countdown  Timers  at  ARTCC 
+++  =  Never  used  product 


FIGURE  5. 2. 1-1,  PRODUCT  UTILITY 
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During  the  interviews,  comments  were  received  concerning  the 
unavailability  of  certain  products  like  windshear,  microburst, 
and  gust  front  when  either  TDWR  or  LLWAS  was  down.  The  lack  of 
an  operational  TDWR  at  the  start  and  finish  of  the  MEM  DEMVAL 
contributed  to  this  problem.  Also,  there  were  several  occasions 
at  MCO  when  lightning  temporarily  disabled  the  LLWAS  system. 
These  problems  are  directly  associated  with  the  individual 
interfacing  sensors;  they  are  not  problems  that  are  inherent  to 
the  ITWS. 

Table  5. 2. 1-2  presents  the  mean  responses  to  questionnaire 
responses  on  timeliness  (product  update  and  display  rate)  and 
ease  of  use.  The  means  are  separated  by  position  and  location. 
Product  update  rate  refers  to  the  speed  that  the  products  are 
updated  with  the  most  recent  sensor  data  (e.g.,  ASR-9 
precipitation  update  rate  is  approximately  30  seconds).  Display 
rate  refers  to  the  response  speed  of  user  requested  display 
changes,  as  presented  in  table  5.2. 1-2.  The  total  mean  rating 
for  product  update,  display  rate,  and  ease  of  use  clearly  meet 
the  maximum  rating  of  2.75. 

Sixty-nine  percent  (45)  of  65  interview  participants  rated 
product  availability  of  the  ITWS  as  good.  Contrary  comments 
referred  to  unavailability  of  TDWR  products  when  the  TDWR  was 
inoperable  in  MEM,  and  the  lack  of  LLWAS  data  when  the  MCO  LLWAS 
was  struck  by  lightning. 

TABLE  5. 2. 1-2.  QUESTIONNAIRE  RESPONSES  TO  PRODUCT  SPEED 

AND  EASE  OF  USE 


II  ORLANDO 

MEMPHIS 

TOTAL 

— 

TOWER 

ARTCC 

TOWER 

ARTCC 

FUNCTION 

SUP/TMC 

CIC 

SUP/TMC 

SUP 

CIC 

SUPTTMC 

UPDATE  RATE 

m 

2.38 

DISPLAY  RATE 

1.83 

1.25 

1^1 

2.38 

1.93 

EASE  OF  USE 

1.34 

1.97 

1.13 

1  1.60 

2,38 

1.78 

5.2.2  Interpretation/Understandabilitv  COI . 

Are  the  ITWS  products  displayed  without  the  need  for  further 
meteorological  interpretation?  Is  the  displayed  information 
easily  understandable  for  the  end-user? 

Responses  to  the  questionnaire  regarding  interpretation/ 
understandability  are  as  follows: 

Do  daylight  conditions  affect  the  readability  of  the  display?  If 
ves.  please  explain. 

There  were  33  responses  to  this  question  from  the  MEM  and  MCO 
Tower /TRACON.  The  responses  were  based  on  the  ITWS  SD  not  the 
TDWR  SD.  (See  section  3.2.2.)  No  responses  were  solicited  from 
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the  ARTCCs,  since  the  operating  environment  is  not  affected  by 
lighting  conditions.  Seventy-six  percent  (25)  of  the  responses 
stated  that  daylight  conditions  did  not  effect  the  readability  of 
the  display.  Eight  respondents  stated  that  glare  and  brightness 
in  the  tower  affected  the  readability  of  the  display,  especially 
at  a  distance. 

Do  nighttime  conditions  affect  the  readability  of  the  display? 

If  yes.  Please  explain. 

There  were  33  responses  to  this  guestion  from  the  MEM  and  MCO 
Tower/TRACON;  all  of  the  responses  to  this  question  stated  that 
readability  of  the  display  is  not  affected  by  nighttime 
conditions . 

Were  there  any  colors  that  you  had  difficulty  differentiating 
between?  If  ves.  what  color  changes  do  you  suggest? 

There  were  41  responses  to  this  question.  Eighty-three  percent 
(34)  of  the  respondents  reported  that  there  were  no  colors  that 
were  difficult  to  differentiate  between.  Color  deficient 
respondents  indicated  minimal  color  differentiation  problems. 

Those  who  responded  "yes"  stated  there  was  some  confusion  among 
the  Microburst  and  level  five  and  six  Precipitation  product 
colors  (which  are  various  shades  of  red) ,  as  well  as  the  Gust 
Front  and  Storm  Extrapolated  Position  product  colors  (purple  and 
light  blue,  respectively). 

Related  questions  asked  during  the  structured  interviews 
included: 

How  interpretable  or  understandable  was  the  meteorological 
information  vou  received? 

There  were  66  responses  to  this  question.  Ninety-five  percent 
(63)  of  the  responses  to  this  question  indicated  that  the 
meteorological  information  was  very  easy  to  understand  and 
interpret,  especially  after  training. 

Did  vou  require  assistance  in  interpreting  the  meteorological 
information? 

There  were  66  responses  to  this  question.  Seventy-four  percent 
(49)  of  the  respondents  indicated  that  they  required  no 
assistance  in  interpreting  the  meteorological  information.  Those 
who  did  state  they  required  some  assistance,  added  that  the 
assistance  was  minimal  or  unnecessary  after  some  familiarity  with 
the  system.  Two  of  the  respondents  stated  that  they  required 
continued  assistance  in  Interpreting  the  meteorological 
information,  even  after  training  and  familiarity. 
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How  would  you  rate  the  time  It  took  to  interpret  meteorological 
information? 

There  were  66  responses  to  this  question.  Ninety-one  percent 
(60)  of  the  respondents  stated  that  the  time  it  took  to  interpret 
the  meteorological  information  was  minimal.  Other  responses  (6) 
included  "average"  or  "acceptable"  in  terms  of  rating 
interpretation  time. 

How  distinguishable  were  the  weather  products  from  each  other? 

There  were  66  responses  to  this  question.  Seventy  percent  (46)  of 
the  respondents  stated  that  the  weather  products  were 
distinguishable  from  each  other. 

The  remaining  30  percent  (20)  related  to  confusion  between  the 
Windshear  and  Storm  Motion  products,  the  Gust  Front  and  SEP 
products,  and  the  Precipitation  and  Microburst  products.  There 
were  also  problems  noted  in  distinguishing  between  level  five  and 
six  precipitation  and  distinguishing  products  at  selected  ranges 
over  30  nm. 

Rate  the  effectiveness  of  weather  information  overlays  on  runways 
and  corridor  locations. 

There  were  46  responses  to  this  question  from  the  MEM  and  MCO 
Tower/TRACON.  Responses  were  not  solicited  from  the  ARTCCs  since 
they  do  not  deal  directly  with  runways  and  terminal  corridors . 
Eighty-three  percent  (38)  of  the  respondents  indicated  that 
weather  information  overlays  on  runways  and  corridor  locations 
were  effective,  especially  for  planning  air  traffic  flow, 
transitions,  arrival /departure  changes,  etc.  Seeing  the  weather 
relative  to  the  airport  and  other  landmarks  as  well  as  user 
control  of  some  of  the  overlays  was  also  deemed  positive. 

Comments  (8)  were  received  to  improve  the  effectiveness  of  the 
weather  information  overlays.  These  included:  keeping  overlays 
over  the  runways  but  not  over  corridors;  and  using  overlays 
similar  to  maps  that  are  already  used.  Other  comments  suggested 
the  inclusion  of  more  defined  range  markings  and  detailed 
airspace  maps.  (The  overlays  are  a  site  adaptable  parameter;  the 
overlays  used  during  the  DEMVAL  were  designed  based  on  input 
received  from  ITWS  User  Group  members  from  Orlando  and  Memphis.) 

Informal  discussions  with  ARTCC  personnel  indicated  that  for  the 
first  time  they  had  a  system  that  showed  them  exactly  where  storm 
cells  were  relative  to  airport  landmarks  and  runways.  This  aided 
them  in  anticipating  storm  arrival  and  departure  at  the  airport. 
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5.2.3  Workload  COI . 


Does  the  ITWS  reduce  perceived  controller  workload  during  adverse 
weather  conditions  in  the  terminal  area? 

Mean  workload  ratings  (figure  4.6. 1.2-1  ten-point  scale,  10  = 
highest  workload,  1  =  lowest  workload)  are  given  in  table  5. 2. 3-1. 
Figure  5. 2. 3-1  depicts  the  overall  workload  rating  of  all  tasks 
for  ITWS  and  non-ITWS  conditions.  Figure  5. 2. 3-2  shows  the 
workload  ratings  by  task.  Figures  5. 2. 3-3  and  5. 2. 3-4  show  the 
workload  rating  by  task  for  each  DEMVAL  site. 

Results  of  the  MCH  Workload  Scale  Indicate  that  perceived  user 
workload  is  significantly  decreased  when  using  the  ITWS,  as  seen 
in  figures  5. 2. 3-1  through  5. 2. 3-4  and  table  5. 2. 3-1. 
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*The  lower  the  workload  rating,  the  lower  the  workload  demand. 


Memphis  Orlando 

Location 


FIGURE  5. 2. 3-1.  WORKLOAD  RATINGS  FOR  NON-ITWS  AND  ITWS 

CONDITIONS  AS  A  FUNCTION  OF  LOCATION 


Task  1  -  Planning  air  traffic  flow 
Task  2  •  Planning  airspace  use 
Task  3  -  Decisions  on  rerouting  aircraft 
T ask  4  -  Avoidance  of  adverse  weather 
Task  5  -  Choosing  runways  (Tower  only) 

Task  6  -  Changing  runways  (Tower  only) 

FIGURE  5. 2. 3-2.  OVERALL  WORKLOAD  RATINGS  FOR  NON-ITWS 

AND  ITWS  CONDITIONS  AS  A  FUNCTION 
OF  TASK 
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TABLE  5. 2. 3-1.  MEAN  WORKLOAD  RATINGS 


II  MCO/ZJX  MEM/ZME  T< 

OTAL 

NON-ITWS 

ITWS 

NON- 

ITWS 

ITWS 

NON- 

ITWS 

ITWS 

TASK  1 

4.51 

2.27 

4.86 

2.72 

4.67 

2.47 

TASK  2 

4.73 

2.19 

5.03 

2.93 

4.87 

2.52 

TASK  3 

5.05 

2.24 

5.10 

2.63 

5.07 

2.41 

TASK  4 

4.78 

2.11 

5.07 

2.63 

4.91 

2.34 

TASKS 

3.96 

1.92 

4.14 

2.36 

4.04 

2.13 

TASK  6 

4.92 

2.00 

4.27 

2.50 

4.63 

2.23 

OVERALL 

4.77 

2.18 

4.76 

2.70 

4.77 

2.41 

Task  1 
Task  2 
Task  3 
Task  4 
Task  5 
Task  6 


Planning  air  traffic  flow 
Planning  airspace  use 
Decisions  on  rerouting  aircraft 
Avoidance  of  adverse  weather 
Choosing  runways  (tower  only) 
Changing  runways  (tower  only) 


Mean  Workload  Rating  Mean  Workload  Rating 


FIGURE  5. 2. 3-3. 


WORKLOAD  RATINGS  FOR  NON-ITWS  AND  ITWS 
CONDITIONS  AS  A  FUNCTION  OF  TASK  FOR 
MEMPHIS  TOWER  AND  ARTCC 


The  lower  the  workload  rating,  the  lower  the  workload  demand. 
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Task  1  -  Planning  air  traffic  flow 
Task  2  •  Planning  airspace  use 
Task  3  -  Decisions  on  rerouting  aircraft 
Task  4  -  Avoidance  of  adverse  weather 
Task  5  -  Choosing  runways  (T ower  only) 
Task  6  -  Changing  runways  (Tower  only) 


FIGURE  5. 2. 3-4. 


WORKLOAD  RATINGS  FOR  NON-ITWS  AND  ITWS 
CONDITIONS  AS  A  FUNCTION  OF  TASK  FOR 
ORLANDO  TOWER  AND  JACKSONVILLE  CENTER 
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5 ♦2. 4  Manaoement/Plannlno  COI. 


Do  the  ITWS  products  enhance  supervisor  effectiveness  in  traffic 
planning/management  during  adverse  weather  conditions  in  the 
terminal  area  (i.e.,  traffic  flow)?  Is  terminal  airspace  planned 
and  runways  utilized  more  efficiently? 

Data  used  to  answer  this  COI  are  in  the  form  of  responses  to 
questions  regarding  use  of  the  ITWS  to  perform  specific 
management  tasks .  Graphical  presentation  of  these  data  was  not 
meaningful . 

How  has  the  ITWS  affected  vour  situational  awareness  of 
Terminal  weather? 

There  were  65  responses  to  this  question.  Eighty  percent 
( 52 )  of  the  respondents  indicated  that  the  ITWS  had 
positively  affected  situational  awareness  of  Terminal 
weather.  Many  respondents  stated  that  in  enhancing 
awareness,  preplanning  was  Improved  as  well  as  coordination 
between  the  Tower/TRACON  and  ARTCCs.  The  remaining 
respondents  indicated  that  awareness  was  not  improved  since 
they  were  already  aware  of  terminal  weather  via  the  ASR-9 
radar  and  reports  from  pilots. 

What  effect  did  the  ITWS  have  on  reduclng/increasino  in¬ 
trail  separation? 

There  were  60  responses  to  this  question.  Forty-two  percent 
(25)  of  the  respondents  indicated  that  the  ITWS  had  a 
positive  effect  on  reducing/increasing  in-trail  separation, 
that  it  was  particularly  useful  in  preplanning,  decision¬ 
making,  and  predicting  airspace  openings  and  closings. 
Fifty-eight  percent  (35)  of  the  users  commented  that  the 
ITWS  had  no  effect  on  in-trail  separation. 

What  products  did  you  use  (for  in-trail  separation)? 

For  those  who  stated  that  the  ITWS  had  an  effect  on 
reducing/increasing  in-trail  separation,  36  responded  to 
this  question.  Products  they  reported  using  most  often  for 
in-trail  separation  included:  Gust  Front,  Precipitation, 
Storm  Motion,  Storm  Extrapolated  Position,  Microburst,  and 
Windshear. 

What  effect  did  the  SEP  Product  have  on  estimating  impact 
times  of  fast  or  slow  moving  storms? 

There  were  64  responses  to  this  question.  Eighty-four 
percent  (54)  responded  that  the  SEP  product  had  a  positive 
effect  on  estimating  impact  times  of  fast  or  slow  moving 
storms.  It  was  reported  to  enhance  preplanning  and  provide 
accurate  estimations  of  weather  effecting  air  traffic 
control  operations. 


36 


Ten  percent  (6)  reported  that  the  SEP  had  no  effect  on 
estimating  impact  times  of  fast  or  slow  moving  storms;  that 
it  was  unnecessary  since  storm  vectors  are  sufficient;  that 
traffic  would  be  rerouted  around  significant  weather  anyway; 
and  that  it  only  adds  clutter  to  the  display.  Four 
respondents  indicated  that  they  had  never  used  it  or  would 
not  use  it  for  estimating  impact  times  of  storms . 

What  effect  did  the  storm  motion  product  have  in  planning 
runway  and  airspace  configuration  changes? 

There  were  66  responses  to  this  question.  Seventy-nine 
percent  (52)  of  the  responses  indicated  that  the  storm 
motion  product  was  very  effective  in  planning  runway  and 
airspace  configuration  changes.  Generally/  respondents  felt 
that  this  was  a  valuable  tool  for  planning  operations  and 
making  timely  decisions. 

The  remaining  21  percent  responded  that  this  product 
afforded  little  or  no  effect  in  planning  runway  and  airspace 
configurations  or  had  never  used  the  product  for  planning. 

How  did  the  oust  front  and  wind  shift  products  affect  runway 
management? 

There  were  46  responses  to  this  question  from  the  MEM  and 
MCO  Tower/TRACON.  No  responses  were  solicited  from  the 
ARTCCs,  since  they  do  not  directly  deal  with  runway 
management.  Seventy-eight  percent  (36)  of  the  respondents 
indicated  that  the  Gust  Front  and  Wind  Shift  products  were 
very  effective  for  runway  management.  Many  indicated  that 
this  was  the  best  product. 

The  remaining  twenty-two  percent  (10)  of  the  respondents 
stated  that  the  Gust  Front  and  Wind  Shift  products  had 
little  or  no  effect  on  runway  management.  They  also  noted 
that  LLWAS  information  was  sufficient. 

If  you  observed  a  tornado  icon  or  alert,  then  what  effect 
did  the  Tornado  product  have  in  routing  aircraft  awav  from 
tornado  activity? 

There  were  30  responses  to  this  question.  Only  one 
respondent  stated  that  they  would  use  the  tornado  product  to 
reroute  aircraft  away  from  tornado  activity. 

The  overall  response  to  this  question  indicated  that  for  the 
most  part,  air  traffic  would  be  rerouted  around  intense 
weather  activity  regardless  of  the  presence  of  a  tornado. 
Thus,  they  felt  there  was  little  use  for  the  product. 
Additionally,  it  was  noted  that  the  information  was  not 
timely  enough.  (Tornadoes  are  capable  of  traveling 
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significant  distances  during  the  5  minutes  required  to 
update  NEXRAD  tornado  data . ) 

How  did  the  ITWS  precipitation  products  effect  vour  planning 
of  traffic  flow  patterns? 

There  were  44  responses  to  this  question  from  the  MEM  and 
MCO  Tower/TRACON.  No  responses  were  solicited  from  the 
ARTCCs,  but  informal  discussions  indicated  that  they  felt 
these  products  to  be  useful  for  traffic  flow  planning. 

Eighty-four  percent  (37)  of  the  respondents  indicated  that 
using  the  ITWS  Precipitation  products  increased 
effectiveness  while  planning  traffic  flow  patterns.  It  was 
cited  as  an  important  product  in  terms  of  ease  of 
discriminating  levels  of  precipitation  and  use  for  planning 
air  traffic  flow  since  aircraft  will  generally  not  fly 
through  precipitation  level  three  and  above. 

The  remaining  seven  respondents  felt  that  this  product  was 
redundant  and  not  effective  in  planning  air  traffic  flow 
since  this  was  already  accomplished  with  the  ASR-9. 

How  did  the  terminal  weather  text  message  product  affect 
vour  situational  awareness? 

There  were  31  responses  to  this  question  from  the  MEM  and 
MCO  Tower/TRACON.  Only  two  respondents  stated  that  the  text 
message  product  positively  effected  their  situational 
awareness. 

The  overall  response  to  this  question  indicated  little  or  no 
use  for  the  terminal  weather  text  message  product  relative 
to  its  effectiveness  for  situational  awareness.  Those  who 
did  use  the  product  found  that  looking  at  the  display  itself 
did  more  for  their  situational  awareness  than  reading  the 
Text  Message. 

What  effect  did  the  terminal  winds  product  have  on  traffic 
management  ( i . e . .  at  the  arrival  and  departure  gates  and  at 
turn-on  to  final  approach)? 

There  were  17  responses  to  this  question  from  the  MEM 
Tower/TRACON.  No  responses  were  solicited  from  the  ARTCCs 
or  MCO  Tower/TRACON. 

Fifty-nine  percent  (10)  of  the  respondents  referred  to  the 
usefulness  of  the  terminal  winds  products  to  anticipate 
tailwinds,  allow  them  to  use  speed  control  more  effectively, 
and  to  plan  final  turn-ons  for  arrivals.  Remaining 
responses  indicated  that  the  product  had  little  impact  on 
operations  and  was  rarely  used. 
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What  effect  did  the  storm  cell  information  have  In  planning 
traffic  flow? 

Forty-seven  percent  (20)  of  the  43  respondents  indicated 
that  using  the  Storm  Cell  Information  product  increased 
their  effectiveness  while  planning  by  providing  controllers 
information  to  determine  pilot-preferred  routes  (around 
adverse  weather)  and  aided  them  in  vectoring  aircraft 
accordingly. 

Twelve  of  the  remaining  respondents  indicated  that  the  storm 
cell  information  was  nice  to  know  but  not  of  any  real  use 
for  air  traffic  operations.  The  remaining  respondents 
experienced  no  effect  from  using  the  product  or  they  did  not 
use  it. 

How  did  the  storm  cell  information  product  affect  your 
situational  awareness? 

Seventy-one  percent  (29)  of  the  41  respondents  to  this 
question  indicated  that  the  storm  cell  information  product 
positively  affected  situational  awareness  and  that  this 
awareness  enhanced  air  traffic  flow  operations.  Other 
responses  indicated  that  although  the  information  was  nice 
to  know,  it  had  minimal  impact  on  situational  awareness. 

5.2.5  Interfacing  Sensor  Availability  COI. 

Does  the  unavailability  of  interfacing  systems /sensors  adversely 
affect  the  ITWS  operations? 

As  stated  in  section  5  and  in  appendices  C  and  D,  there  was  a 
high  rate  of  availability  of  TDWR,  ASR-9,  and  NEXRAD  data  to  the 
ITWS  during  the  DEMVAL.  The  Memphis  DEMVAL  was  started  without 
the  TDWR  available,  and  was  completed  with  the  TDWR  inoperable. 
While  windshear,  microburst,  and  gust  front  products  were  not 
available  at  these  times,  other  ITWS  products  remained  available. 
AT  personnel  continued  to  have  the  ability  to  use  the 
precipitation  products,  storm  motion  and  extrapolated  position, 
AP,  lightning,  and  degraded  terminal  winds  and  storm  cell 
information. 

There  were  several  times  during  the  Orlando  DEMVAL  that  the  LLWAS 
was  struck  by  lightning,  rendering  it  inoperable.  At  these 
times.  Runway  Winds  (on  the  RBDT)  were  unavailable.  However, 
microburst  and  windshear  alerts,  which  are  TDWR  products, 
continued  to  be  available. 
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5.2.6  Meteorological  Performance. 


Are  detections  and  false  alarm  rates  acceptable  to  air  traffic 
users? 

The  ITWS  Operational  Requirements  Document  specifies 
meteorological  performance  requirements  in  terms  of  probabilities 
of  detection  and  false  alarm  for  the  microburst  detection, 
microburst  prediction,  and  gust  front  detection  products. 

Measures  of  acceptable  performance  for  the  other  products  are 
described  in  the  appropriate  subsections  below.  These 
performance  levels  were  the  goal  for  ITWS  meteorological  product 
performance  during  the  DEMVAL. 

Results  from  individual  product  performance  evaluations  are 
provided  in  the  following  sections.  Details  of  the  analyses  and 
analysis  methods  along  with  examples  are  presented  in  appendix  B. 
Data  descriptions  as  well  as  the  times,  dates,  and  sites  used  in 
each  analysis  are  also  provided  in  appendix  B. 

All  analyses  were  based  on  limited  quantities  of  data.  Caution 
is  advised  when  interpreting  the  results  presented.  The  intent 
of  the  ACT-320  analysis  was  to  "spot  check"  the  performance  of 
each  of  the  products.  A  more  comprehensive  analysis  of  the 
DEMVAL  product  performance  will  be  reported  by  MIT/LL. 

5.2.6. 1  Short-Range  Precipitation. 

The  ITWS  short-range  precipitation  product  is  based  on  the  ASR-9 
weather  channel.  Two  problems  exist  with  the  ASR-9  weather 
channel.  It  is  susceptible  to  anomalous  propagation  (AP),  and  it 
can  be  insensitive  to  precipitation  very  close  to  the  radar  site. 
These  problems  were  evident  in  the  ASR-9  data  when  the  data  were 
compared  with  NEXRAD  and/or  TDWR  data  (see  section  5. 2. 6. 2  and 
appendix  B,  section  B.l).  In  spite  of  these  problems,  the 
performance  of  this  product  was  acceptable. 

5. 2. 6. 2  Anomalous  Propagation  (AP)  Edition  Precipitation. 

A  qualitative  analysis  in  which  the  AP  edited  precipitation 
product  was  compared  with  unedited  ASR-9  reflectivity  and  NEXRAD 
composite  reflectivity  was  employed  in  the  evaluation  of  this 
product.  Results  of  the  analysis  indicate  that  AP  was 
appropriately  reduced  or  eliminated.  Additional  Information  is 
provided  in  appendix  B. 

The  evaluated  data  set  provided  an  example  of  a  limitation  of  the 
current  system.  This  limitation  affects  the  availability  of 
NEXRAD  data  for  AP  editing  close  to  the  NEXRAD  site.  The  NEXRAD 
is  affected  by  ground  clutter  to  a  range  of  approximately  20 
kilometers.  There  is  also  a  small  area  within  10  kilometers  of 
the  NEXRAD  called  the  cone  of  silence.  Within  this  area  the 
NEXRAD  can  detect  precipitation  only  in  the  lowest  part  of  the 
atmosphere.  AP  editing  is  not  possible  in  these  areas.  This 
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limitation  is  not  a  problem  if  the  affected  NEXRAD  area  is 
outside  the  ASR-9  coverage  area.  The  affected  NEXRAD  area  is 
within  the  ASR-9  coverage  area  in  both  Memphis  and  Orlando. 


5 . 2 . 6 . 3  Long-Range  Precipitation . 

The  long-range  precipitation  product  was  observed  and  found  to  be 
comparable  to  the  short-range  precipitation  product  where  the  two 
products  coincide.  The  ITWS  long-range  product  was  compared 
visually  with  the  NEXRAD  base  reflectivity  data  display,  and  was 
determined  to  be  consistent  with  the  base  data.  The  performance 
of  this  product  was  acceptable. 

5. 2. 6. 4  Storm  Motion. 

Results  from  the  data  analysis  of  the  storm  motion  product 
indicate  that  it  was  capable  of  providing  an  estimation  of  cell 
motion.  This  motion  was  consistent  with  the  leading  edges 
projected  by  the  SEP  product.  The  performance  of  this  product 
was  acceptable. 

5. 2. 6. 5  Storm  Extrapolated  Position  fSEPK 

The  effectiveness  of  the  SEP  product  in  extrapolating  storm 
leading  edge  positions  10  and  20  minutes  in  the  future  is 
dependent  upon  the  degree  of  storm  evolution  (i.e.,  growth  and 
decay) .  The  current  SEP  algorithm  was  not  designed  to  account 
for  storm  evolution;  it  projects  the  motion  of  a  storm  cell  of  a 
particular  size  and  shape,  and  expects  the  cell  to  retain  the 
same  size  and  shape  for  as  many  as  20  minutes.  In  20  minutes  an 
evolving  storm  could  grow  such  that  its  leading  edge  is  in  a 
location  quite  different  from  the  location  projected  by  the  SEP 
product.  A  decaying  storm  for  which  leading  edges  were  projected 
may  not  exist  20  minutes  in  the  future. 

The  data  used  in  the  analysis  were  found  to  contain  evolving 
storms.  (Examples  of  evolving  storms  are  provided  in  appendix  B, 
section  B.4.)  This  determination  led  to  the  definition  and  use 
of  three  performance  statistics: 

a.  The  probability  of  making  a  correct  extrapolation  using 
all  extrapolations,  including  those  for  evolving  cells  (PcEl)/ 

b.  The  probability  of  making  a  correct  extrapolation, 
excluding  extrapolations  for  evolving  cells  (Pcee)/ 

c.  The  probability  that  a  cell  would  be  affected  by  growth 
and  decay  (Pea) • 

For  the  SEP  product  performance  analysis,  a  total  of  204  10-  and 
20-minute  extrapolation  were  examined.  Computed  statistics  are 
shown  in  table  5. 2. 6. 5-1. 
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TABLE  5. 2. 6. 5-1.  STORM  EXTRAPOLATED  POSITION  PRODUCT  EVALUATION 

STATISTICS. 


10  min 

20  min 

Number  of  extrapolations 

204 

204 

Number  affected  by  cell 

67 

97 

evolution  (growth/decay/merglng) 

Number  of  hits 

100 

70 

Number  of  misses 

37 

37 

PCEI 

.49 

.34 

PCEE 

.73 

.65 

Pea 

.33 

.47 

The  Peas  indicate  that  after  10  minutes  nearly  one-third  of  the 
extrapolations  were  affected  by  growing  or  decaying  storms.  This 
increased  to  almost  one-half  after  20  minutes. 

There  was  also  a  problem  which  caused  erroneous  leading  edges  and 
extrapolations  under  certain  circumstances.  These  erroneous 
extrapolations  are  included  in  table  5. 2. 6. 5-1  as  "misses.”  More 
information  about  these  problems  is  presented  in  appendix  B, 
section  B.4. 

5. 2. 6. 6  Microburst /Windshear  Detection. 

In  order  to  evaluate  microburst  and  windshear  detection 
performance,  the  following  statistics  were  computed;  Probability 
of  Detection  (Pd)  and  Probability  of  False  Alarm  (PpA) 
microbursts,  and  Pd  and  PpA  windshears.  These  statistics 

were  computed  for  each  site  separately  and  in  combination  for  a 
subset  of  the  total  microburst/windshear  detections. 

Microbursts  and  windshears  as  detected  by  the  ITWS,  were  examined 
and  compared  with  the  a  truth  set  of  base  TDWR  velocity  (raw 
sensor)  data  as  described  in  appendix  B,  section  B.5.  ITWS 
detections  corresponded  favorably  with  the  truth  set;  there  were 
no  missed  detections,  and  only  one  false  detection.  This  false 
detection  was  in  a  region  of  noisy  velocity  data. 

Table  5. 2. 6. 6-1  presents  the  statistics  for  microburst  (MB) 
detection  and  includes  the  Pd  and  PpA-  Similar  statistics  for 
windshear  (WS)  detection  are  given  in  table  5. 2. 6. 6 -2.  Separate 
and  combined  statistics  are  given  for  both  Memphis  and  Orlando. 
For  the  data  set  examined,  the  microburst/windshear  product 
scored  perfectly  in  terms  of  detection  (Pd  =  1-00  for  Memphis  and 
Orlando);  i.  e.,  all  microbursts  in  Memphis  and  Orlando  were 
detected,  as  were  all  windshears.  PpA^  were  well  below  the 
requirement  of  0.10. 
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TABLE  5. 2. 6. 6-1.  MICROBURST  DETECTION  STATISTICS  (>30  KT) . 


#MB 

#MB 

tCorrect  MB 

Pd 

PpA 

(Truth) 

Detections 

Detections 

Memohis 

6/3 

2 

2 

2 

6/22 

7 

7 

7 

7/4 

1 

1 

1 

Total 

10 

10 

10 

1.00 

0.00 

Orlando 

7/28 

4 

4 

4 

8/9 

1 

1 

1 

8/15 

7 

7 

7 

Total 

12 

12 

12 

1.00 

0.00 

Total 

(MEM+MCO) 

22 

22 

22 

1.00 

0.00 

TABLE  5. 

2 . 6 . 6-2 . 

WINDSHEAR  DETECTION  STATISTICS 

(>15  KT, 

<30  KT) . 

#ws 

#WS 

#Correct  WS 

Pd 

PpA 

( Truth ) 

Detections 

Detections 

Memphis 

6/3 

18 

18 

18 

6/22 

16 

16 

16 

7/4 

6 

6 

6 

Total 

40 

40 

40 

1.00 

0.00 

Orlando 

7/28 

23 

23 

23 

8/9 

10 

10 

10 

8/15 

16 

17 

16 

Total 

49 

50 

49 

1.00 

0.02 

Total 

(MEM+MCO) 

89 

90 

89 

1.00 

0.01 

Results  indicate  that  the  microburst  detection  product  performed 
very  well.  However,  these  results  should  be  taken  with  some 
caution,  as  a  relatively  small  portion  (about  6  hours)  of  the 
demonstration  data  were  analyzed. 
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5. 2. 6. 7  Microburst  Prediction. 


Two  probabilities  were  computed  for  microburst  prediction.  One 
was  the  probability  of  prediction  (Pp) .  The  second  statistic  was 
the  probability  of  false  alarm  (Pfa)  windshear  events  with 

strength  greater  than  15  kt. 

Table  5. 2. 6. 7-1  presents  the  microburst  prediction  statistics  for 
the  data  set  examined.  The  PpA®  related  to  this  analysis  were 
computed  from  the  numbers  contained  in  table  5. 2. 6. 7-2. 

Cumulative  statistics  for  all  days  at  each  site  are  given  as  well 
as  combined  statistics  for  all  days  and  both  sites. 

The  analysis  indicated  that  the  microburst  prediction  product 
performed  well  when  it  attempted  to  predict  an  event.  In  all 
instances  of  missed  microbursts,  the  product  did  not  attempt  to 
predict  the  event. 

The  ORD  states  that  the  Pp  shall  be  greater  than  0.50.  The  Pp 
for  Memphis  (0.69)  exceeded  the  requirement,  but  the  Orlando  Pp 
(0.23)  was  much  lower  (less  than  half)  of  the  required  Pp.  The 
combined  Memphis  and  Orlando  Pp  (0.48)  was  only  slightly  below 
the  requirement.  MIT/LL  has  performed  a  more  comprehensive 
analysis  that  may  indicate  better  prediction  performance. 

TABLE  5. 2. 6. 7-1.  MICROBURST  PREDICTION  PRODUCT  STATISTICS, 

PROBABILITY  OF  PREDICTION  (>25  KT) . 


Number 

of  Events 

Truth 

Predicted 

Pp 

Memphis  (MEM) 

6/3 

5 

5 

6/22 

8 

3 

7/4 

3 

3 

Total 

16 

11 

0.69 

Orlando (MCO) 

7/28 

10 

2 

8/9 

0 

0 

8/15 

3 

1 

Total 

13 

3 

0.23 

Total 

(MEM+MCO) 

29 

14 

0.48 
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TABLE  5. 2. 6. 7-2. 


MICROBURST  PREDICTION  PROBABILITY  OF 
FALSE  ALARM  (>15  KT) . 


1 - 

Number  of 

Number  of  Correct 

Pfa 

Predictions 

Predictions 

Memohis  fMEM) 

6/3 

26 

26 

6/22 

5 

5 

7/4 

27 

27 

Total 

58 

58 

0.00 

Orlando (MCO) 

7/28 

4 

4 

8/9 

0 

- 

8/15 

17 

17 

Total 

21 

21 

0.00 

Total 

(MEM+MCO) 

79 

79 

0.00 

The  statistics  shown  in  table  5. 2. 6. 7-2  indicate  PpA®  0.00. 
There  were  no  false  alarms  generated  by  the  microburst  prediction 
product  during  the  analysis  periods.  This  perfect  score 
satisfies  the  ORD  requirement  for  Pp;^  (<  0.1). 

5 . 2 . 6 . 8  Gust  Front  Detection . 

Probabilities  of  detection  (PpL)  false  alarm  (Pfal)'  based  on 

length,  were  used  to  determine  the  performance  of  the  gust  front 
detection  product.  Additional  information  is  contained  in 
appendix  B,  section  B.7. 

The  resultant  P^l  ^FAL  the  gust  front  detection  product 

were  0.79  and  0.04,  respectively.  These  meet  the  ORD  accuracy 
requirements  for  P^l  (>  0.7)  and  PpAL  (<0.1). 

5. 2. 6. 9  Terminal  Winds. 

Results  of  the  qualitative  analysis  indicated  that  the  gridded 
winds  of  the  terminal  winds  product  were  consistent  with  the 
velocity  fields  of  the  NEXRAD  and  TDWR  in  the  areas  where  radar 
data  were  provided.  Additional  details  are  found  in  appendix  B. 

5.2.6.10  Storm  Cell  Information. 

The  storm  cell  information  product  reports  echo  tops  and  the 
presence  of  hail,  lightning,  and  mesocyclones  within  selected 
storm  cells.  Each  component,  when  assigned  to  a  particular  cell, 
is  referred  to  as  an  "attribute."  Assessments  of  the  capability 
of  the  product  to  associate  echo  tops,  hail,  lightning,  and 
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mosocyclones  with  a  storm  cell  follow  In  section  5. 2. 6. 10.1 • 
Assessments  of  the  performance  of  the  hail  detection/  echo  topS/ 
and  mesocyclone  detection  algorithms  appear  in  the  remaining 
sections . 

5.2.6.10.1  Attribute  to  Cell  Association. 

Hail.  Results  from  the  analysis  indicate  that  hail  algorithm 
detections  were,  for  the  most  part,  properly  associated  with 
storm  cells.  There  were/  however/  missed  associations  (i.e./ 
hail  algorithm  detections  that  the  product  failed  to  associate 
with  a  storm  cell).  Appendix  E,  section  B.9/  contains  a 
discussion  on  these  missed  hail  associations. 

Lightning.  Lightning  flashes  appeared  to  be  scattered  adequately 
over  the  map  so  that  each  cell  containing  lightning  received  the 
lightning  attribute.  There  were/  however/  many  flashes  that  were 
not  associated  with  storm  cells  (see  appendix  B,  figures  B8-B13). 

Mesocyclone .  Three  missed  mesocyclone  associations  are  discussed 
in  appendix  E,  section  B.9.  In  these  cases /  there  were  no  cells 
with  which  to  associate  the  mesocyclones . 

Echo  tops .  Analysis  results  indicate  that/  for  most  cases 
examined/  maximum  echo  tops  were  found  and  associated  with  the 
appropriate  cell.  There  were/  however/  cases  for  which  maximum 
echo  tops  were  outside  the  cell  boundary.  These  situations  are 
discussed  in  appendix  B,  section  B.9.1. 

5.2.6.10.2  Hail. 

The  NWS  compiles  reports  of  hail  sightings.  There  were  22 
sightings  of  hall  in  excess  of  0.75  inches  in  the  Memphis  ITWS 
coverage  area  during  June  1994.  For  17  of  these  sightings / 
there  were  corresponding  hail  product  detections.  For  two  of 
these  sightings /  the  ITWS  hail  product  did  not  detect  the  NWS 
hail  event.  For  the  remaining  three  NWS  sightings,  the  ITWS  was 
not  operational  at  the  time  of  occurrence.  See  appendix  B, 
section  B.9. 2  for  details  of  the  analysis. 

5.2.6.10.3  Echo  Tops. 

Early  in  the  demonstration  at  Memphis  DEMVAL  it  was  found  that 
the  echo  tops  provided  by  the  ITWS  echo  tops  product  were  lower 
than  those  of  the  NEXRAD  by  as  much  as  5000  feet.  MIT/LL  found 
an  error  in  their  software  and  corrected  it.  Thereafter,  the 
echo  tops  product  appeared  to  be  consistent  with  the  echo  tops 
reported  by  the  NEXRAD. 

5.2.6.10.4  Mesocyclone. 

Data  adequate  for  evaluating  the  mesocyclone  product  were 
unavailable  at  the  time  of  this  report.  The  NWS  does  not  compile 
reports  of  public  sightings  of  mesocyclones;  so  an  analysis 
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similar  to  the  hail  product  analysis  was  not  possible.  MIT/LL  or 
NSSL  will  issue  a  report  on  the  performance  of  this  product. 

5.2.6.11  Lightning  Warning. 

The  lightning  warning  box  product  was  evaluated  in  real-time  by 
comparing  it  with  the  Storm  Cell  Information  product.  Whenever 
the  lightning  alert  box  changed  to  yellow,  the  SCI  boxes  for 
storm  cells  within  20  nm  of  the  airport  center  were  checked  for 
the  lightning  attribute.  Likewise,  when  the  lightning  attribute 
appeared  in  an  SCI  box  for  a  storm  cell  within  20  nm  of  the 
airport  center,  the  lightning  box  was  checked  to  ensure  it  had 
changed  to  yellow.  Observations  indicated  that  the  warning  box 
product  registered  lightning  strikes  as  expected. 

5.2.6.12  Tornado. 

A  total  of  79  tornadoes  were  determined  to  have  occurred  in  the 
Memphis  ITWS  coverage  area  in  June  1994.  These  79  tornadoes 
constitute  the  truth  set  for  this  analysis  of  the  tornado  product 
and  were  determined  from  examination  of  base  TDWR  and  NEXRAD 
velocity  data.  Of  these  79  tornadoes,  only  42  were  detected  by 
the  ITWS  tornado  product.  This  resulted  in  a  Pp  of  0.53.  There 
were  eight  false  detections,  with  a  of  0.16.  These 
statistics  are  presented  in  table  5.2.6.12-1. 

The  ORD  does  not  specify  performance  requirements  for  the  tornado 
product.  However,  the  Pp^  of  0.16  is  close  to  the  value  of  0.10, 
the  maximum  allowable  Pp^y  specified  for  the  ITWS  microburst  and 
gust  front  products  in  the  ORD. 


TABLE  5.2.6.12-1.  TORNADO  DETECTION  STATISTICS 

(JUNE  1994,  MEMPHIS) 


Hits 

Misses 

False  Detections 

Pd 

PPA 

42 

37 

8 

0.53 

0.16 

NSSL  is  conducting  an  evaluation  of  the  tornado  product  using 
DEMVAL  data  and  results  from  experiments  they  are  conducting  in 
other  areas  of  the  United  States .  Their  studies  are  expected  to 
result  in  refinement  of  the  tornado  algorithm  and  improvement  of 
performance,  and  in  particular,  the  Pp. 

5.2.6.13  Terminal  Weather  Text  Message. 

The  terminal  text  message  was  compared  with  the  SD.  The  product 
appeared  to  be  consistent  with  the  SD.  Locations  were  within 
roughly  ±  1.0  nm. 
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6.  CONCLUSIONS. 


The  conclusions  that  were  reached  are  based  on  the  results  of  the 
questionnaires,  interviews,  workload  scales,  and  meteorological 
analysis  as  they  relate  to  the  Critical  Operational  Issues  (COI). 
The  results  were  analyzed  from  an  air  traffic  (AT)  perspective. 

The  responses  received  from  the  questionnaires,  interviews  and 
informal  discussions  were  not  always  consistent.  Similarly,  they 
did  not  always  correlate  with  the  product  performance  analysis. 
Products  AT  personnel  consider  marginally  useful  may  prove  to  be 
much  more  useful  to  other  users  (e.g.,  flight  standards,  systems 
capacity,  the  airlines,  etc.). 

6.1  COMPLETION  CRITERIA. 

Sufficient  adverse  weather  was  experienced  during  the  Memphis  and 
Orlando  Demonstration/Validation  Phase  (DEMVAL)  Operational  Test 
and  Evaluation  (OT&E)  from  which  to  properly  evaluate 
meteorological  performance  and  Integrated  Terminal  Weather  System 
(ITWS)  product  suitability.  A  sufficient  number  of  AT  personnel 
utilized  the  ITWS  during  adverse  weather  conditions. 

6.2  CRITICAL  OPERATIONAL  ISSUES. 

Each  of  the  COIs  that  were  evaluated  for  DEMVAL  are  listed  below 
in  boldface  type.  The  conclusions  are  contained  in  the  following 
paragraphs . 

1.  Are  the  ITWS  products  useful  during  operationally  significant 
weather  in  terms  of  their  availability,  timeliness,  and 
suitability  for  air  traffic  use?  Are  detection  and  false  alarm 
rates  acceptable  to  users? 

The  results  presented  in  sections  5.2.1  and  5.2.6  support  the 
conclusion  that  for  the  Demonstration  Phase,  the  ITWS  products 
were  useful  in  terms  of  availability,  timeliness,  and  suitability 
for  air  traffic  use.  Some  products  were  rated  higher  than 
others,  but  all  were  rated  as  enhancing  user  job  performance.  No 
products  were  rated  as  hindering  job  performance,  although  the 
utility  of  some  products  (lightning,  terminal  weather  text 
message,  and  tornado)  was  rated  as  marginal.  The  lightning  and 
tornado  products  may  be  more  useful  to  systems  capacity,  airlines 
dispatch,  and  ramp  personnel,  while  the  terminal  weather  text 
message  product  may  be  useful  to  pilots,  in  the  form  of  the 
terminal  weather  information  for  pilots  (TWIP) . 

There  were  some  users  that  responded  that  product  availability 
was  not  acceptable.  In  these  instances,  unavailability  was  due 
to  the  interfacing  sensor  availability,  not  a  function  of  ITWS 
product  unavailability.  Terminal  Doppler  Weather  Radar  (TDWR) 
unavailability  in  Memphis  prohibited  the  generation  of  windshear 
and  gust  front  products;  Low  Level  Windshear  Alert  System  (LLWAS) 
lightning  damage  caused  product  unavailability  in  Orlando  on 
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several  occasions.  Otherwise  ITWS  products  were  available  at  all 
times.  Interfacing  sensor  availability  remains  a  risk. 

The  timeliness  aspect  of  this  COI  was  based  on  user  responses  on 
display  rates  and  update  rates  of  selectable  products  and  display 
options.  Respondents  rated  the  update  speed  of  the  products  and 
the  speed  at  which  the  selected  products  were  displayed  as 
frequently  enhancing  their  performance.  A  number  of  respondents 
did  state  that  the  update  rate  for  the  range  and  anomalous 
propagation  (AP)  editor  products  was  slow.  Additionally/  half  of 
the  respondents  wanted  a  faster  update  speed  for  the  products 
overall. 

The  ITWS  products  were  rated  as  useful  and  suitable  for  AT 
personnel  use.  Availability  and  timeliness  of  the  products  were 
satisfactory/  but  require  improvement.  The  ITWS  DEMVAL  operating 
software  was  written  in  four  different  computer  languages  (e.g. 

C/  C++/  LISP/  FORTRAN);  processing  speed  was  not  optimized. 
Production  software  using  a  single  higher  order  programming 
language  should  minimize/eliminate  processor  induced  delays  in 
product  display.  In  some  cases /  users  confused  ITWS  product 
update  rates  with  the  update  rates  of  the  input  sensors. 

Based  on  analysis  of  meteorological  data  collected  during  the 
DEMVAL/  it  was  determined  that  the  ITWS  products  performed  to  an 
acceptable  level  relative  to  probability  of  detection  and  false 
alarm  rates  and  other  performance  levels.  Some  of  the  products 
require  further  work  in  order  to  meet  the  performance  standards 
of  the  Operational  Requirements  Document  (ORD) /  prior  to  formal 
OT&E  testing.  Conclusions  specific  to  certain  products  are 
listed  below: 

a.  The  Airport  Surveillance  Radar  Model  9  (ASR-9)  had 
difficulty  detecting  precipitation  in  the  area  near  the  radar  due 
to  radar  insensitivity.  This  is  an  ASR-9  radar  problem/  and  not 
within  the  scope  of  the  ITWS  DEMVAL. 

b.  There  is  a  high  probability  that  the  ITWS  storm 
extrapolated  position  product  can  be  affected  by  the  growth  and 
decay  of  storm  cells.  This  product  was  not  designed  to  account 
for  growth  and  decay. 

c.  The  microburst  detection  product  performed  very  well. 

The  Pp  for  the  product  was  1.00  for  the  combined  Memphis  and 
Orlando  cases /  and  the  0.02  for  Orlando  and  0.00  for 

Memphis.  The  PpA  the  combined  Memphis  and  Orlando  cases  was 

0.01. 


d.  The  microburst  prediction  product  performance  was 
acceptable  when  it  attempted  to  predict  microbursts.  However,  it 
did  not  always  attempt  to  predict  all  microbursts  that  did  occur. 
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e.  The  gust  front  product  scored  a  Pj^l  of  0.79  and  a  Pp^L 
of  0.04.  These  numbers  met  the  Operational  Requirements  Document 
(ORD)  requirements  of  0.7  and  0.1. 

f.  Not  all  hail  and  mesocyclone  events  were  associated  with 
storm  cells /  and  were  not  listed  as  storm  cell  attributes  by  the 
storm  cell  Information  product 

g.  Initially,  ITWS  echo  tops  were  being  reported  as  lower 
than  those  being  reported  by  the  Next  Generation  Weather  Radar 
(NEXRAD);  this  was  also  found  to  be  due  to  association  problems. 
The  highest  echo  tops  were  found  to  be  outside  the  storm  cell 
contour.  This  problem  was  corrected  during  the  DEMVAL; 
subsequent  analysis  indicated  acceptable  performance. 

h.  There  is  a  20-km  radius  circle  centered  on  the  NEXRAD 
wherein  data  are  sparse  or  unusable  due  to  the  NEXRAD 's  cone  of 
silence  and  ground  clutter.  There  is  no  AP  editing  capability  in 
this  area.  If  this  area  is  coincident  with  the  ASR-9  coverage 
area,  no  AP  editing  will  take  place.  This  is  a  known  problem 
with  the  NEXRAD,  and  was  outside  the  scope  of  the  ITWS  DEMVAL. 

i.  The  tornado  product  is  a  replacement  for  the  current 
NEXRAD  tornado  vortex  signature  product.  Based  on  analyses  by 
Massachusetts  Institute  of  Technology/Lincoln  Labs  (MIT/LL)  and 
ACT-320,  the  product  scored  a  Pp  of  0.53  and  a  Ppj^  of  0.16. 

Given  what  is  available  in  terms  of  the  tornado  product,  this 
performance  is  satisfactory;  improvements  are  still  required. 

j .  Slight  discrepancies  in  the  terminal  weather  text 
message  (TWTM)  product  can  be  explained  by  realizing  that  the 
text  message  may  have  been  based  on  a  radar  scan  taken  just  prior 
to  the  scan  which  the  meteorologist  used  for  comparison. 

2.  Are  the  ITWS  products  displayed  without  the  need  for  further 
meteorological  interpretation?  Is  the  displayed  information 
easily  understandable  for  the  end-user? 

Overall,  the  ITWS  products  satisfied  this  COI  for  this  stage  of 
testing,  in  terms  of  interpretability  and  understandability. 
Results  derived  from  data  collected  in  relation  to  this  COI 
support  this  conclusion.  AT  personnel  found  the  meteorological 
information  easy  to  understand  and  interpret.  After  training  and 
familiarity  with  the  system,  AT  personnel  considered  the  time 
required  to  interpret  the  meteorological  information  as  minimal. 
Proper  training  will  continue  to  assure  a  high  degree  of 
understanding  of  ITWS  products  as  they  are  displayed  on  the 
Geographic  Situation  Display  (SD). 

One-third  of  the  respondents  found  some  difficulties  in 
retrieving  information  from  the  display  during  daylight 
conditions.  There  was  some  difficulty  in  distinguishing 
information  on  the  display  due  in  part  to  the  similarity  of  some 
colors  used  for  different  products.  Noted  examples  are  storm 
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extrapolated  position  versus  gust  front  and  the  similarity  in 
color  between  levels  five  and  six  precipitation  and  their 
similarity  of  the  microburst  icon. 

3.  Does  the  ITWS  reduce  (perceived)  controller  workload  during 
adverse  weather  conditions  in  the  terminal  area? 

Results  from  the  workload  scale  data  analysis  indicate  that  the 
ITWS  significantly  reduced  perceived  controller  workload  during 
the  DEMVAL,  in  the  presence  of  adverse  weather  conditions.  This 
was  demonstrated  in  the  Tower/Terminal  Area  Approach  Control 
(TRACON)  and  Air  Route  Traffic  Control  Center  (ARTCC) 
environments  for  both  Orlando  and  Memphis.  Across  all  job  tasks 
evaluated,  workload  was  rated  significantly  lower  when  utilizing 
ITWS  than  when  not  utilizing  the  ITWS. 

4.  Do  the  ITWS  products  enhance  supervisor  effectiveness  in 
traffic  planning/management  during  adverse  weather  conditions  in 
the  terminal  area  (i.e.,  traffic  flow)?  Is  terminal  airspace 
planned  and  runways  utilized  more  efficiently? 

Traffic  management,  planning,  and  flow,  as  well  as  terminal 
airspace  and  runway  management  were  enhanced  by  the  ITWS  products 
as  indicated  by  the  structured  interviews .  Situational  awareness 
and  the  degree  of  coordination  between  the  Tower/TRACON  and  the 
ARTCC  was  increased,  creating  a  smoother  transition  of  aircraft 
from  the  ARTCC  to  the  Tower/TRACON.  Both  long-  and  short-range 
precipitation  products  were  found  to  be  effective  for  planning 
air  traffic  flow  of  arriving  and  departing  aircraft. 

Use  of  the  storm  motion  product  increased  the  effectiveness  of 
planning  runway  and  airspace  configuration  changes  and  valuable 
for  planning  operations  and  timely  decision  making.  The  gust 
front  and  windshear  products  were  also  found  to  be  increase  the 
effectiveness  of  runway  management. 

The  terminal  winds  product  was  assessed  in  terms  of  its 
effectiveness  for  traffic  management.  No  conclusion  can  be  drawn 
on  the  effect  the  terminal  winds  product  had  on  traffic 
management  due  to  the  disparity  of  the  responses,  and  the  limited 
data  set  (Memphis  only) . 

Several  products  neither  enhanced  air  traffic  planning  and 
situational  awareness  nor  hindered  job  performance.  In 
particular,  storm  cell  information  and  terminal  weather  text 
message  were  felt  to  have  insignificant  impact  in  enhancing 
situational  awareness  and  traffic  planning.  The  tornado  product 
was  not  effective  for  rerouting  air  traffic  away  from  a  tornado, 
since  pilots  automatically  avoid  areas  of  severe  weather. 
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5.  Does  the  unavailability  of  interfacing  systems /sensors 
adversely  affect  the  ITWS  operations? 

All  input  sensors  were  available  for  the  vast  majority  of  the 
DEMVAL  period  in  Orlando;  The  TDWR  was  the  only  sensor  with 
measurable  unavailability  in  Memphis.  Lightning  strikes  rendered 
the  LLWAS  unavailable  on  several  occasions  in  Orlando. 

Sensor  unavailability  did  not  adversely  affect  ITWS  operations 
during  the  DEMVAL.  During  times  of  individual  sensor 
unavailability,  AT  personnel  were  able  to  continue  to  use  the 
ITWS,  given  the  high  availability  rate  of  the  input  sensors 
during  the  DEMVAL.  Some  of  the  input  sensor  data  were  received 
via  MIT/LL,  not  the  operational  interface,  compounding  the 
assessment.  Since  input  sensor  availability  was  high,  a 
quantitative  assessment  of  ITWS  system  performance  with  reduced 
capability  cannot  be  given. 

The  ITWS  contained  usable  products  even  without  the  availability 
of  some  of  the  interfaces .  This  is  due  to  the  redundancy  that 
exists  with  the  major  interfaces  (e.g.,  TDWR  or  NEXRAD  used  to 
display  0-50  nm  range  precipitation  upon  loss  of  ASR-9  Weather 
Channel,  LLWAS  III  used  to  generate  microburst  detections  and 
RBDT  runway  alerts,  TDWR  used  to  edit  ASR-9  AP  in  case  of  NEXRAD 
failure) .  Therefore,  it  can  be  concluded  that  the  ITWS  is  less 
susceptible  than  other  systems  to  interface  induced  outages . 

Table  3.2-1  of  the  ITWS  ORD  illustrates  product  availability  in 
relation  to  sensor  availability. 

6 . 3  PRODUCT  IMPROVEMENTS . 

Current  RBDT  configurations  do  not  display  all  runway  alerts; 
they  only  display  alerts  for  their  respective  runways . 

Controllers  unaware  of  alerts  present  on  adjacent  runways  may 
inadvertently  vector  aircraft  into  these  alerts. 

A  countdown  timer  indicating  gust  front  airport  impact  time  would 
be  useful.  AT  personnel  would  not  have  to  estimate  time  to 
impact  based  on  the  gust  front  lines  on  the  SD. 

Suggestions  for  improvements  of  ITWS  products  were  minimal;  they 
are  presented  in  appendix  A.  These  suggestions  are  pertinent  to 
the  ultimate  design  of  the  ITWS  products;  many  of  them  have  been 
discussed  at  the  ITWS  Users  meetings  and  will  be  implemented  in 
future  ITWS  prototypes. 

6 . 4  USER  INTERFACE . 

Many  of  the  conclusions  listed  in  the  following  paragraphs  are 
based  on  results  reported  in  appendix  A.  Overall  confidence  in 
the  displayed  products  was  very  high,  but  responses  received  from 
AT  users  indicate  that  some  of  them  question  the  accuracy  of  the 
microburst  and  windshear  products.  This  is  contrary  to  the 
results  of  the  microburst  and  windshear  detection  product 
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evaluation  presented  in  sections  5. 2. 6. 5  and  5. 2. 6. 6/  which 
indicate  excellent  detection  and  false  alarm  performance. 

There  were  some  users  that  suggested  larger  SDs.  At  this  time, 
the  ITWS  is  a  strategic  (supervisory)  tool,  not  for  use  by 
controllers.  Consideration  should  be  given  to  making  the  ITWS  a 
tactical  tool  for  use  by  air  traffic  controllers  since  they  use 
the  ITWS  when  they  are  acting  as  controllers  in  charge  (CIC). 

The  ITWS  would  be  beneficial  to  controllers  as  well  as 
Supervisors  and  Traffic  Management  Coordinators  (TMC) . 

The  effect  the  Lightning  product  has  on  influencing  a 
supervisor's  decision  to  turn  on  backup  generators  was  assessed. 
Overall  responses  Indicated  that  the  Lightning  product  was  not 
needed  for  this  task,  due  to  the  existence  of  the  uninterruptible 
power  supply  (UPS);  however,  not  all  Airport  Traffic  Control 
Towers  (ATCT)  and  ARTCCs  may  have  a  UPS.  In  addition,  the 
Lightning  product  is  a  useful  product  with  respect  to  ramp 
operations  and  airline  dispatch.  The  benefits  to  these  groups 
were  not  assessed  by  ACT-320  since  Flights  Standards  and  Systems 
Capacity  were  not  official  sponsors  at  the  time  of  the  DEMVAL. 
Benefits  to  these  groups  were  assessed  by  the  Benefits  Assessment 
Team. 

The  terminal  weather  text  message  product  had  minimal  impact  on 
the  number  of  requests  from  pilots  for  details  about  convective 
weather  in  the  terminal  area  or  on  radio  traffic  during  the 
DEMVAL.  Most  respondents  were  unaware  of  its  existence. 

The  effect  of  switching  from  the  long-  to  short-ranges  of  the 
precipitation  products  was  inconclusive,  due  to  the  disparity  of 
the  responses.  Half  of  the  respondents  related  that  movement  was 
easy,  whereas  the  other  half  noted  that  the  movement  was  slow  and 
cumbersome. 

The  selection  process  for  some  of  the  product  can  be  cumbersome; 
too  many  steps  are  involved.  The  speed  at  which  some  selected 
products  and  options  are  updated  is  too  slow  (  e.g.,  changing  the 
short-range  precipitation  range,  changing  from  short-  to  long- 
range  precipitation) . 

7 .  RECOMMENDATIONS . 

Based  on  the  results  of  the  Integrated  Terminal  Weather  System 
(ITWS)  1994  Demonstration  Validation  (DEMVAL)  and  the  conclusions 
that  were  reached,  ACT-320  recommends  that  the  ITWS  program 
proceed  to  Key  Decision  Point  3  (KDP-3).  Some  of  the 
recommendations  that  are  made  correspond  to  results  and  comments 
that  fall  within  more  than  one  category  (e.g..  Critical 
Operational  Issues  (COIs),  product  improvement,  user  interface); 
these  recommendations  are  listed  one  time. 
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7.1  COMPLETION  CRITERIA. 


No  reconunendations  are  given. 

7.2  CRITICAL  OPERATIONAL  ISSUES. 

a.  Continue  to  monitor  the  ITWS  input  sensor  interfaces  and 
their  development.  This  can  be  accomplished  by  continuing  to 
collect  data  during  subsequent  DEMVALs  and  monitoring  Interface 
Requirement  Document  (IRD)  development  via  Interface  Control 
Working  Group  (ICWG)  meetings,  both  with  the  program  offices  that 
direct  the  interfacing  sensors  and  the  prime  contractor. 

Reliable  input  sensor  interfaces  should  provide  the  ITWS  with  a 
high  rate  of  availability. 

b.  To  assure  continued  user  satisfaction  with  ITWS  product 
availability,  usefulness,  timeliness,  and  suitability,  continue 
data  collection  and  evaluation  at  future  DEMVALs  and  continue  the 
ITWS  AT  User  Group  meetings.  Minimize  product  update  rate  and 
quantify  the  number  of  required  steps  to  change  product  options. 

c.  Ensure  that  at  future  DEMVALs:  (1)  input  sensor 
interfaces  are  hardened  as  much  as  possible  prior  to  the  start  of 
DEMVAL;  (2)  sites  have  commissioned  input  sensors  in  order  to 
ensure  higher  reliability/availability  of  input  data;  and  (3) 
system  software  should  be  optimized  as  much  as  possible  to  give 
the  user  a  realistic  system  to  operate. 

d.  Use  the  Terminal  Doppler  Weather  Radar  (TDWR) 
reflectivity  to  generate  the  ITWS  precipitation  product  for  the 
5-nm  range  in  order  to  negate  the  effects  of  Airport  Surveillance 
Radar  Model  9  (ASR-9)  insensitivity  in  this  region.  Investigate 
the  possibility  of  including  this  radar  data  into  the  ASR-9 
mosaic  for  selected  ranges  greater  than  5  nm. 

e.  Modify  the  storm  extrapolated  position  (SEP)  algorithm 
to  account  for  the  effects  of  growth  and  decay  of  storm  cells. 

f.  Continue  to  evaluate  the  microburst  and  wlndshear 
algorithms,  particularly  the  microburst  prediction  algorithm. 

The  accuracy  of  the  prediction  algorithm  was  acceptable  but  the 
probability  of  prediction  performance  needs  Improvement. 

Document  pilot  reports  and  compare  to  the  ITWS  information. 

g.  Collect  TDWR  data  as  the  radars  are  installed.  The  data 
collected  at  these  sites  should  be  used  to  improve  the  accuracy 
of  microburst  and  windshear  detection  and  prediction  algorithms . 

h.  Improve  storm  cell  information  product  association 
techniques  for  echo  tops,  hail,  and  mesocyclone  products.  Storm 
cells  may  need  to  be  redefined  with  extended  association  areas. 

i.  Investigate  the  feasibility  of  using  the  TDWR  to  perform 
anomalous  propagation  (AP)  editing  wherever  the  ASR-9  coverage 
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falls  within  the  Next  Generation  Weather  Radar  (NEXRAD)  cone  of 
silence  and  ground  clutter  affected  areas. 

j .  Continue  analysis  of  the  improved  tornado  algorithm  and 
include  in  future  ITWS  product  suites. 

k.  Develop  a  comprehensive  training  program  in  order  to 
maintain  a  high  level  of  user  understanding  of  the  ITWS  products 
and  display. 

l.  A  situation  display  (SD)  with  maximum  contrast  and 
brightness  control  is  required  in  order  to  ensure  maximum 
visibility  of  displayed  products.  Tests  should  be  conducted  to 
evaluate  SD  visibility  under  all  anticipated  lighting  conditions. 
Ensure  that  the  SD  is  optimally  located  in  order  to  maximize 
product  and  display  visibility  under  all  conditions. 

m.  Further  investigation  of  color  discernabillty  is 
required  to  minimize  confusion  between  products. 

n.  Further  evaluation  is  required  in  regards  to  the  effects 
of  the  terminal  winds,  terminal  weather  text  message,  and  tornado 
products  on  traffic  planning  and  management,  as  well  as  non-AT 
users.  A  terminal  winds  evaluation  will  take  place  in  Orlando 
during  February  1995. 

o.  Keep  the  SDs  in  the  Air  Route  Traffic  Control  Centers 
(ARTCCs)  to  maintain  situational  awareness  and  smooth 
coordination  between  the  ARTCC  and  Tower. 

p.  Conduct  a  controlled  test  of  the  effects  of  sensor 
unavailability  on  the  ITWS. 

q.  Controller  tasks  in  the  ARTCC  and  terminal  environments 
need  to  be  assessed  to  ensure  ITWS  products  provide  all  the 
necessary  information  to  complete  weather  related  job  tasks. 

r.  Evaluate  the  effects  of  ITWS  on  AT  personnel  performance 
in  an  environment  that  has  never  had  ITWS  or  TDWR. 

s.  Monitor  current  product  development  activities  for 
inclusion  as  future  ITWS  products. 

7.3  PRODUCT  IMPROVEMENTS. 

Recommendations  for  product  improvements,  based  on  feedback  from 
interviews  and  questionnaires  include  the  following. 

a.  Modify  the  ribbon  display  terminal  to  present  current 
windshear  and  microburst  warnings  for  all  runway  configurations 
at  all  controller  positions. 

b.  Place  centerfield  wind  information  on  the  top  of  the 
Ribbon  Display. 
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c.  Investigate  the  feasibility  of  including  a  gust  front 
impact  timer  within  the  gust  front  warning  box. 

d.  Investigate  the  feasibility  of  allowing  multiple 
windows  on  the  SD  to  display  either  different  precipitation 
ranges  (e.g.,  200  nm  and  50  nm)  or  different  airports  within  the 
same  Terminal  Radar  Approach  Control  (TRACON)  area.  This  would 
be  particularly  useful  for  multiple  airport  TRACONs  and  ARTCCs 
controlling  multiple  ITWS  airports. 

e.  Investigate  the  inclusion  of  storm  trend  (growth  and 
decay)  within  the  storm  cell  information. 

f.  Investigate  the  inclusion  of  graphical  lightning  data  on 
the  SD. 

g.  Evaluate  additional  product  improvement.  Some  products 
require  low  level  detail  areas  to  be  evaluated  and  refined, 
whereas  other  products  require  high  level  utility  issues  to  be 
addressed  (e.g.,  the  text  on  the  terminal  winds  product,  the 
tornado  product  and  the  effectiveness  of  the  display  to  alert 
controllers,  and  the  density  of  information  displayed). 

7 . 4  USER  INTERFACE . 

User  interface  recommendations  include: 

a.  Streamline  the  product  selection  process;  too  many  steps 
are  required  for  some  processes . 

b.  A  quantitative  requirement  is  needed  for  the  number  of 
steps  required  to  access  products  and  select  options . 
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8 .  ACRONYMS . 


ACT-320 

AFSFO 

AP 

APMT 

AND-420 

AND-460 

ARTCC 

ASOS 

ASR-9 

AT 

ATC 

ATCT 

ATIS 

ATR-400 

AWOS 

CIC 

CO  I 

CWSU 

DEMVAL 

FAA 

FSL 

GFE 

GFI 

ICWG 

IOC 

IRD 

ITWS 

LLWAS 

MAPS 

MB 

MBA 

MCH 

MCO 

MDCRS 

MEM 

MIGFA 

MIT/LL 

NAS 

NATCA 


Weather  Branch 

Airways  Facilities  Sector  Field  Office 
Anomalous  Propagation 
Associate  Program  Manager  For  Test 
TDWR  Program  Office 

Aviation  Weather  Development  Program 

Air  Route  Traffic  Control  Center 

Automated  Surface  Observing  System 

Airport  Surveillance  Radar,  Model  9 

Air  Traffic 

Air  Traffic  Control 

Airport  Traffic  Control  Tower 

Automated  Terminal  Information  System 

Air  Traffic  Requirements  Service,  Program  Sponsor 

Automated  Weather  Observing  System 

Controller  In  Charge 

Critical  Operational  Issue 

Center  Weather  Service  Unit 

Demonstration/Validation 

Federal  Aviation  Administration 

Forecast  Systems  Laboratory 

Government  Furnished  Equipment 

Government  Furnished  Information 

Interface  Control  Working  Group 

Initial  Operational  Capability 

Interface  Requirements  Document 

Integrated  Terminal  Weather  System 

Low  Level  Windshear  Alert  System 

Mesoscale  Area  Prediction  System 

Microburst 

Microburst  Alert 

Modified  Cooper-Harper  Workload  Scale 
Orlando  International  Airport 

Meteorological  Data  Collection  And  Reporting  System 
Memphis  International  Airport 
Machine  Intelligent  Gust  Front  Algorithm 
Massachusetts  Institute  Of  Technology/Lincoln 
Laboratories 

National  Airspace  System 

National  Air  Traffic  Controllers  Association 
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NCP 

NEXRAD 

NLDN 

NSSL 

NWS 

ORD 

OT&E 

OTS 

PPA 

Pd 

PD 

POSH 

Pp 

RBDT 

SCI 

SD 

SEP 

T&E 

TDWR 

TEMP 

TMC 

TMU 

TRACON 

TWTM 

TWIP 

UPS 

VCP 

VIL 

WS 

WSA 

WSFO 

ZJX 

ZME 


NAS  Change  Proposal 

Next  Generation  Weather  Radar,  WSR-88D 

National  Lightning  Data  Network 

National  Severe  Storms  Laboratory 

National  Weather  Service 

Operational  Requirements  Document 

Operational  Test  And  Evaluation 

Out  Of  Service 

Probability  Of  False  Alarm 

Probability  Of  Detection 

Program  Directive 

Probability  Of  Severe  Hail 

Probability  of  Prediction 

Ribbon  Display  Terminal 

Storm  Cell  Information 

Situation  Display 

Storm  Extrapolated  Position 

Test  And  Evaluation 

Terminal  Doppler  Weather  Radar 

Test  And  Evaluation  Master  Plan 

Traffic  Management  Coordinator 

Traffic  Management  Unit 

Terminal  Radar  Approach  Control 

Terminal  Weather  Text  Message 

Terminal  Weather  Information  For  Pilots 

Uninterruptible  Power  Supply 

Volume  Coverage  Pattern 

Vertically  Integrated  Liquid 

Windshear 

Windshear  Alert 

Weather  Service  Forecast  Office 
Jacksonville  ARTCC 
Memphis  ARTCC 
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APPENDIX  A  -  QUESTIONNAIRE  AND  INTERVIEW  RESULTS 

This  appendix  contains  additional  results  from  data  collected  from 
the  questionnaires  and  interviews,  but  not  directly  related  to  the 
COIs . 

Some  of  the  questionnaire  rating  scale  questions  asked  users  to  rate 
aspects  of  the  ITWS  other  than  the  utility  of  the  products.  The 
results  of  these  questions  are  as  follows: 

Identify  and  explain  anv  product  that  was  difficult  to  use. 

There  were  nine  responses  to  this  question.  Six  of  the  respondents 
indicated  that  there  were  no  products  that  were  difficult  to  use. 

Other  users  indicated  that  it  would  be  easier  to  change  the  selection 
steps  from  three  to  one;  that  user  specific  preference  sets  could  be 
devised  to  enhance  ease  of  use  of  the  products;  and  that  changing  the 
cursor  may  help  in  finding  it  on  the  display.  The  selection  process 
has  been  discussed  at  the  ITWS  User  Group  meetings  and  in  all 
likelihood  will  be  modified  to  reduce  the  number  of  steps  required  to 
perfoirm  a  selection. 

Identify  and  explain  anv  selected  product  in  which  the  display  speed 
was  unacceptable. 

There  were  eight  responses  to  this  question.  Four  of  the  respondents 
stated  that  there  were  no  selected  products  in  which  the  display 
speed  was  unacceptable.  Three  stated  that  the  update  rate  was  a 
little  slow  for  the  range  and  AP  editor  selection  products.  One 
responded  "not  applicable." 

Identify  and  explain  anv  product  in  which  the  update  speed  was 
unacceptable. 

There  were  12  responses  to  this  question.  Seven  respondents  reported 
that  the  update  speed  was  acceptable.  The  remaining  comments 
indicated  faster  update  speeds  were  desired  overall,  and  one 
indicated  a  faster  tornado  product  update  speed.  (In  general,  the 
update  rates  of  the  ITWS  products  are  directly  related  to  the  update 
rates  of  the  respective  input  sensors,  and  thus  are  not  changeable 
within  the  ITWS  software.) 

Was  the  5-second  time  out  of  the  AP  editor  product  acceptable? 

Thirty-eight  responses  to  this  question  were  received.  Thirty-five 
of  the  respondents  stated  that  the  AP  editor  product  was  acceptable. 
Of  the  remaining  responses,  only  one  stated  that  a  10-second  timer 
would  be  more  efficient;  and  two  stated  they  had  never  used  it. 

Some  questions  requested  controllers  to  rate  the  readability  of 
various  aspects  of  product  timeliness.  The  following  are  the  mean 
rating  scale  responses  to  these  questions . 
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Rate  the  readability  of  the  text  Information  on  the  terminal 
winds  product.  The  mean  was  2.08,  frequently  enhances  job 
performance. 

Rate  the  density  of  information  when  using  the  storm 
extrapolated  position  product.  The  mean  rating  was  1.99, 
frequently  enhances  job  performance. 

Rate  the  readability  of  the  textual  Information  provided  by  the 
tornado  product .  The  mean  response  was  2.28,  consistent  with  the 
responses  reported  above;  the  readability  of  this  product  does 
enhance  controller  performance.  It  must  be  remembered  that  this 
product  was  only  utilized  on  those  rare  occasions  when  a  tornado 
occurred. 

Other  questions  solicited  specific  information  concerning  suggested 
changes  to  existing  products,  as  listed  below. 

Is  there  any  product  information  in  a  textual  format  that  you 
would  prefer  in  a  graphical  format? 

There  were  34  responses  to  this  question.  Overall,  the 
respondents  stated  that  there  were  no  products  in  a  textual 
format  that  they  would  prefer  in  a  graphical  format.  Three  of 
the  respondents  would  prefer  to  see  lightning  strikes  in  a 
graphical  format. 

Is  there  any  product  information  in  a  graphical  format  that  you 
would  prefer  in  a  textual  format? 

There  were  36  responses  to  this  question.  With  only  two 
exceptions,  each  respondent  stated  that  there  were  no  products 
in  a  graphical  format  that  they  would  prefer  in  a  textual 
format.  One  of  the  respondents  would  prefer  to  see  the 
microburst  product  in  text  with  distance  and  radial  to  aid  in 
identifying  it  Immediately. 

Additional  comments  were  solicited  per  product. 

ITWS  Precipitation  Product. 

There  were  12  responses  to  this  question.  Seven  of  the 
respondents  stated  that  they  had  no  suggestions  for  improving 
the  ITWS  precipitation  product. 

Those  who  had  suggestions  for  improvement  suggested;  a  window 
to  show  a  variable  range  while  watching  a  larger  picture  on  the 
main  screen,  and  a  feature  to  predict  areas  of  building 
precipitation . 

AP  Editor  Product. 

There  were  21  responses  to  this  question.  All  respondents 
stated  that  they  had  no  suggestions  for  improving  the  AP  editor 
product . 
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Long-Range  Precipitation  Product. 


Twenty-one  controllers  responded  to  this  question.  Fourteen 
respondents  stated  that  they  had  no  suggestions  for  improvement 
of  the  long-range  precipitation  product. 

There  were  several  suggestions  for  improvement  which  included: 
the  incorporation  of  more  range  choices;  speeding  the  product 
up;  addition  of  more  fixes  (NAVAIDS);  reduction  of  steps  for 
product  selection  (from  three  to  one);  and  splitting  the  GSD 
screen  out  into  separate  windows,  allowing  views  of  the  long- 
and  short-ranges  simultaneously. 

Storm  Motion  Product. 

Of  the  19  responses  to  this  question,  fourteen  had  no 
suggestions  for  improving  the  storm  motion  product.  The 
remaining  suggestions  for  improvement  included:  display  of  the 
back  side  of  the  storm  to  predict  changes;  and  display  of  the 
storm  motion  product  at  ranges  greater  than  5  knots  past  the  50 
nm  range. 

Storm  cell  information  product  -  Are  there  additional  products 
that  you  would  like  to  be  added? 

There  were  28  responses  to  this  question.  Sixteen  had  no 
suggestions  for  improving  the  storm  cell  information  product. 

Nor  did  they  identify  any  additional  products  that  they  would 
like  to  be  added. 

Suggestions  for  additions  included:  displaying  the  bottom  level 
of  a  storm  cell;  showing  a  trend  in  storm  intensity  (increase, 
decay,  etc.);  and  eliminating  the  storm  top  information.  (Storm 
tops  frequently  occur  above  the  terminal  airspace  ceiling  of 
16,000  feet.) 

Storm  Extrapolated  Position  Product. 

There  were  22  responses  to  this  question.  Sixteen  had  no 
suggestions  for  improving  the  storm  extrapolated  position 
product . 

Respondents  who  had  suggestions  stated  that  the  color  contrast 
between  gust  front  and  the  storm  extrapolated  position  should  be 
improved.  It  was  also  suggested  that  the  product  wasn't 
necessary  and  that  storm  motion  vectors  were  sufficient. 
Typically,  storms  move  at  much  slower  speeds  in  Orlando  than 
they  do  in  Memphis,  hence  storm  cells  are  easier  to  track 
without  this  product.  The  usefulness  of  this  product  will 
depend  on  the  prevailing  characteristics  of  local  storm 
movement . 
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Lightning  Product. 


There  were  26  responses  to  this  question.  Fourteen  respondents 
had  no  suggestions  for  improving  the  lightning  product. 

Those  who  had  suggestions  wanted  to  see  lightning  strikes 
displayed  per  location  (much  like  the  3M  Scope).  Another 
suggested  that  the  lightning  product  not  be  a  part  of  the  final 
package  since  it  was  unnecessary. 

Microburst  Detection  Product. 

There  were  25  respondents  to  this  question.  Fourteen  had  no 
suggestions  for  improving  the  microburst  detection  product. 

There  were  several  suggestions  for  improvement  of  the  microburst 
detection  product.  Many  respondents  stated  that  the  microburst 
color  should  be  changed  to  avoid  confusion  with  levels  of 
weather  (similarity  to  level  5  and  6  precipitation). 

Several  suggestions  also  centered  around  the  concern  that 
microburst  alerts  should  be  displayed  on  all  RDTs  regardless  of 
runway  configuration.  This  has  subsequently  been  discussed  by 
the  ITWS  User  Group;  possible  solutions  will  be  evaluated  during 
future  prototype  evaluations. 

This  information  may  have  an  impact  on  those  departures  that  do 
not  fly-out  on  a  straight  line  and  may  be  impacted  by  weather 
affecting  the  other  runway.  Other  suggestions  included  having 
the  product  more  accessible  to  the  flight  data  position  for  the 
ATIS,  and  feedback  from  pilots  when  the  microburst  alert  is 
issued. 

Terminal  Winds  Product. 

There  were  eight  responses  to  this  question.  This  question  was 
not  applicable  to  Orlando  Tower/TRACON  and  Jacksonville  ARTCC, 
since  the  product  was  not  available  to  them. 

Four  of  the  respondents  had  no  suggestions  for  improving  the 
terminal  winds  product.  The  only  suggestion  for  improvement  of 
the  terminal  winds  product  was  to  make  winds  aloft  information 
easier  to  understand. 

Do  you  have  any  suggestions  for  improving  the  oust  front 
product? 

There  were  21  responses  to  this  question.  Fifteen  of  the 
respondents  had  no  suggestions  for  improving  the  gust  front 
product.  Suggestions  for  improvement  centered  on  the  inclusion 
of  a  count-down  mode  on  the  warning  panel  to  indicate  gust  front 
impact  time.  This  also  has  been  discussed  by  the  User  Group  and 
will  be  implemented  and  evaluated  during  future  prototype 
operations . 
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ATIS  Countdown  Timer  Product. 


There  were  17  responses  to  this  question.  Fourteen  of  the 
respondents  had  no  suggestions  for  improving  the  ATIS  countdown 
timer  product. 

Suggestions  for  improving  the  ATIS  countdown  timer  product 
included:  a  turn-off  command  in  the  event  it  is  accidentally 

turned  on;  an  automatic  tie  to  the  ATIS;  and  better  placement  in 
the  tower  for  easier  accessibility. 

Tornado  Product. 

There  were  25  responses  to  this  question.  Eleven  of  the 
respondents  had  no  suggestions  for  improving  the  tornado 
product.  Eight  of  the  respondents  stated  that  they  had  never 
seen  the  product  operationally. 

Those  who  had  suggestions  for  improving  the  tornado  product 
generally  stated  that  the  update  rate  was  too  slow;  tornado 
information  was  too  infrequent;  and  that  the  range  for  tornado 
warnings  on  the  Ribbon  Display  should  be  farther  than  10  miles. 
The  tornado  update  rate  is  dictated  by  the  scan  rate  of  the 
NEXRAD  radar;  the  maximum  range  of  the  tornado  alert  on  the  RDT 
is  a  site  adaptable  parameter. 

Additional  comments  and  suggestions  were  solicited  from  the 
questionnaire  respondents,  as  shown  below. 

Please  provide  any  other  comments  or  suggestions  you  may  have 
regarding  the  products,  display,  and/or  system. 

There  were  37  responses  to  this  request  for  comments  and 
suggestions.  Twenty-five  of  the  respondents  provided  comments 
to  this  general  question  regarding  the  ITWS  products,  display, 
and/or  system.  The  respondents  stated  that  the  system  was 
excellent;  beneficial;  useful;  enhanced  safety;  a  valuable  tool 
for  the  TMU;  the  best  tool  in  traffic  management;  invaluable; 
enhances  traffic  management;  the  best  product;  etc. 

Others  stated  that  the  reliability  must  be  improved;  the  ribbon 
display  information  was  incorrect;  and  that  overall  the 
information  was  only  occasionally  useful.  Since  ITWS 
reliability  was  nearly  100  percent,  increased  input  sensor 
reliability  should  improve  perceived  ITWS  reliability. 

Eleven  of  the  respondents  provided  suggestions  for  improving  the 
ITWS.  These  include:  splitting  the  display  screen  into  windows 
to  view  both  the  long  and  close  range;  making  the  displays 
accessible  to  everyone  (controllers,  flight  data,  etc.); 
eliminating  the  "Done"  box  after  a  user  selection;  and  placing 
the  center  field  wind  information  on  the  top  line  of  the  ribbon 
display. 
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How  would  you  rate  movement  through  the  different  display 
screens? 

There  were  65  responses  to  this  question.  Approximately  half 
(33)  of  those  who  responded  stated  that  movement  through  the 
different  display  screens  was  easy  or  good. 

Thirty-two  of  the  respondents,  however,  did  not  rate  movement 
through  the  different  displays  positively.  Overall,  the 
comments  were  similar  in  expressing  that  the  movement  was  too 
slow,  cumbersome,  and  difficult. 

If  you  used  lightning  products  to  turn  on  back-up  generators, 
what  effect  did  the  lightning  product  have  on  vour  decision  to 
switch  to  back-up  generator  power? 

There  were  17  responses  to  this  question.  Comments  were  not 
solicited  from  the  ARTCCs.  Only  three  respondents  actually 
stated  that  they  would  use  the  lightning  product  to  turn  on 
backup  power.  Others  responded  that  they  were  aware  of 
lightning  and  didn't  need  help  in  deciding  when  to  turn  on  the 
generators.  The  overall  response  to  this  question  was  negative 
due  to  the  existence  of  the  Universal  Power  Supply  (UPS) .  "The 
UPS  would  go  on  automatically  as  soon  as  lightning  was  within  20 
nm.  Therefore,  the  product  is  unnecessary  for  this  task.  Also, 
it's  easy  enough  to  tell  if  lightning  is  nearby  -  just  look  out 
the  window." 

How  confident  were  you  in  the  accuracy  of  the  displayed 
products,  i.e.,  warnings,  storm  movements,  etc.? 

There  were  65  responses  to  this  question.  Seventy-eight  percent 
of  the  responses  (51)  were  positive.  Confidence  in  the 
displayed  products  was  high  for  the  majority  of  respondents. 

Although  expressing  confidence  in  most  of  the  products,  some 
respondents  (14)  did  have  some  reservations  about  a  few  specific 
products.  Those  products  identified  most  often  were  windshear 
and  microburst.  It  was  noted  that  the  products  would  sometimes 
become  activated  despite  reports  from  pilots  or  other  indicators 
that  there  was  no  significant  wind  activity.  It  was  suggested 
that  the  algorithms  for  these  products  may  be  too  sensitive. 

This  is  contrary  to  the  results  of  the  microburst  and  windshear 
product  performance  evaluation  presented  in  sections  5. 2. 6. 5  and 
5. 2. 6. 6,  which  Indicate  excellent  detection  and  false  alarm 
probabilities . 

How  did  the  microburst  products  affect  vour  degree  of  confidence 
during  landing  and  take-off  operations? 

There  were  44  responses  to  this  question.  Comments  were  not 
solicited  from  the  ARTCCs.  Fifty-four  percent  of  the 
respondents  (25)  stated  that  the  microburst  products  had  a 
positive  effect  on  their  degree  of  confidence  during  landing  and 


A-6 


take-off  operations.  This  was  especially  true  in  terms  of 
warning  pilots  of  microburst  activity. 

Thirteen  indicated  that  confidence  was  not  high;  that  the 
product  did  not  accurately  portray  what  was  actually  happening; 
that  the  algorithm  was  too  sensitive;  and  that  bad  weather 
conditions  would  deter  landing  and  take-off  operations  anyway  - 
whether  or  not  a  microburst  was  detected.  Three  reported  not 
using  the  product;  and  three  indicated  that  the  microburst 
product  did  not  effect  their  degree  of  confidence. 

Rate  the  impact  of  the  terminal  weather  text  message  product  on 
the  number  of  requests  from  pilots  for  details  about  convective 
weather  in  the  terminal  area. 


The  mean  overall  score  to  this  question  was  3.07.  This  suggests 
that  there  was  no  significant  impact  of  this  product  on  the 
number  of  requests  from  pilots  for  details  about  convective 
weather  in  the  terminal  area. 

Rate  the  impact  of  the  terminal  weather  text  message  product  on 
radio  traffic  in  general. 

The  mean  overall  score  to  this  question  was  3.05.  This  also 
suggests  that  there  was  no  significant  impact  of  the  product  on 
radio  traffic  in  general. 

Provide  any  comments  on  the  terminal  weather  text  message 
demonstration  that  vou  consider  to  be  relevant; 

There  were  36  responses  to  this  question.  Twenty  respondents  (56 
percent)  were  unfamiliar  with  the  product  or  were  unaware  of  any 
impact  it  had  to  air  traffic  control  operations. 

There  were  seven  responses  (19  percent)  from  those  who  had 
experienced  some  Impact  from  pilots  using  the  terminal  text 
product.  Most  of  this  interaction  was  minimal.  Some  of  the 
more  positive  respondents  stated  that:  a  pilot  delayed  approach 
based  on  terminal  text  information;  there  was  more  agreement 
between  the  pilots  and  the  controllers;  and  that  it  assisted  in 
preplanning  for  holding  aircraft.  Less  positive  comments  stated 
that:  the  product  precipitated  questions  from  pilots  about  lack 
of  information  to  the  aircraft;  some  outside  telephone 
communications  increased  with  Northwest  carrier  operations; 
there  were  more  questions  on  microburst;  and  a  pilot  reported  no 
depiction  of  character  graphics  in  the  cockpit. 

Seven  respondents  (19  percent)  made  hypothetical  statements 
about  the  potential  positive  or  negative  impact  of  the  product 
on  air  traffic  control  operations.  Two  of  the  responses  were 
not  relevant  to  the  question. 
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METEOROLOGICAL  PERFORMANCE 


This  appendix  contains  details  of  the  analyses  and  analysis  methods 
used  to  obtain  the  results  presented  in  the  meteorological 
performance  evaluation  section,  section  5«2.6« 

B.l  Short  Range  Precipitation 

The  ITWS  Short  Range  Precipitation  Product  was  observed  to  have 
problems  providing  accurate  measurement  of  cell  precipitation  and 
coverage  near  the  radar.  An  example  of  this  problem  was  a  well- 
defined  windshear  producing  storm  cell  that  could  readily  be  seen 
in  TDWR  data,  but  was  barely  noticeable  in  ASR-9  data.  The  TDWR, 
located  11  nm  south  of  the  airport,  appeared  much  more  sensitive  to 
the  cells  than  the  ASR-9  located  in  the  terminal  area.  At  1830  UTC 
on  July  4,  storm  cells  developed  over  and  near  the  Memphis  terminal 
area.  These  cells  remained  in  the  terminal  area  for  about  one 
hour,  generating  windshear  features  and  then  dissipating. 

Figure  B-1  shows  the  ITWS  GSD  with  the  ASR-9  short-range 
precipitation  product  at  184403.  The  lower  left  panel  of  Figure  B- 
2  shows  TDWR  reflectivity  at  184348.  The  TDWR  panel  indicates 
reflectivity  values  of  40  dBZ  close  to  and  over  the  airport 
runways.  (A  measurement  of  40  dBZ  is  at  the  upper  end  of  the  range 
for  level  two  precipitation.)  Note  that  in  Figure  B-1  there  were 
only  two  squares  of  light  green  (level  one)  in  the  area  where  the 
TDWR  indicated  level  two  precipitation.  The  ASR-9  was  detecting 
only  a  small  portion  of  the  cell.  The  problem  persisted  for  about 
an  hour,  until  the  cells  dissipated. 

B.2  Anomalous  Propagation  (AP)  Edited  Precipitation 

Color  data  plots  (an  example  plot  is  shown  in  Figure  B-3)  used  to 
evaluate  this  product  contained  four  panels: 

1)  Map  of  the  raw,  unedited  ASR-9  reflectivity  (upper  left 
panel) ; 

2)  Map  of  the  NEXRAD  composite  reflectivity  (upper  right); 

3)  Map  of  the  raw,  unedited  ASR-9  reflectivity,  with  echo 
regions  determined  to  be  affected  by  AP  shown  in  black  (lower 
left); 

and 

4)  Map  of  the  AP  edited  precipitation  (lower  right). 
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FIGURE  B-1.  ITWS  GEOGRAPHIC  SITUATION  DISPLAY  (GSD),  JULY  4,  1994,  184403  UTC. 
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Real-time  display  at  Memphis,  TN.  Note  sparse  precipitation  intensity  over  airport  (compare 
with  Fig.  B-2). 
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Lower  left  panel  shows  TDWR  reflectivity  over  airport  (compare  with  Fig.  B-1). 


FIGURE  B-3.  DATA  PLOT,  JUNE  8,  1994,  184029  UTC,  MEMPHIS,  TN. 
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Shows  evaluation  data  for  AP  edited  precipitation  product.  Upper  left  panel  depicts  raw  ASR- 
9  reflectivity.  Upper  right  is  a  composite  of  NEXRAD  reflectivity.  Lower  left  is  the  ASR-9 
precipitation  with  AP  affected  areas  shown  in  black.  The  lower  right  panel  is  the  AP  edited 
precipitation  product. 


The  NEXRAD  composite  reflectivity  map  contains,  at  each  pixel  on 
the  map,  the  highest  NEXRAD  reflectivity  sensed  in  the  vertical 
column  above  that  pixel.  The  NEXRAD  provides  the  composite 
reflectivity  by  performing  a  set  of  successive  scans  while 
increasing  elevation  angles  (this  scan  strategy  is  known  as  a 
Volume  Coverage  Pattern  (VCP)).  The  NEXRAD  completes  a  VCP 
approximately  once  every  five  to  six  minutes .  The  ASR  9 
precipitation  updates  every  30  seconds. 


The  AP  editing  process  was  evaluated  by  comparing  the  raw  ASR~9 
sSn  eldest  in  time  to  the  middle  of  the  VCP  with  the  NEX^ 
composite  reflectivity  map.  This  comparison  provided  a  reference 
ASR-9  map  with  pixels  flagged  as  being  affected  by 
reference  was  first  used  in  conjunction  with  the  next  VCP.  Each 
ASR-9  update  map  during  the  VCP  was  compared  with  the  reference 
from  the  prior  completed  VCP.  Each  update  map  was  eligible  for  AP 
editing  only  at  pixels  that  were  flagged  on  the  reference  ASR-9 
map.  AP  editing  rules  prescribed  that  pixels  on  the  update  map  be 
assigned  lower  precipitation  levels,  or  remain  the  same,  depending 
upon  the  maximum  allowed  reflectivity  (from  the  NEXRAD  composite 
reflectivity  map)  for  the  individual  pixel. 


For  the  AP  edited  precipitation  product,  the  following  data  were 
analyzed: 


Memphis  6/8/94  1840-1903  UTC 

B.3  Storm  Motion 

In  order  to  evaluate  the  data  for  the  storm  motion  product  four- 
panel  plots  were  generated  from  a  time  history  of  storm  cells. 

A  sample  data  plot  used  in  the  evaluation  of  the  Storm  Motion 
product  is  shown  in  Figure  B-4 .  The  top  left  panel  on  the  plot 
contains  the  current  precipitation  product  along  with  the  storm 
motion  (arrow  and  numeric  showing  direction  and  speed)  for  each 
significant  storm.  The  other  panels  contain  data  related  to  the 
Storm  Extrapolated  Position  product  (see  section  B.4),  and  are  not 
relevant  to  the  current  discussion.  The  plots  were  updated 
approximately  ojice  every  two  minutes. 

The  location  of  each  storm  cell  shown  in  the  top  left  panel  of  each 
plot  was  compared  visually  with  the  location  of  the  corresponding 
storm  cell  in  successive  plots.  From  these  comparisons, 
determination  of  motion  for  each  cell  was  made.  A  check  for 
consistency  of  direction  was  made  with  the  extrapolated  leading 
edge  positions,  where  possible. 

Data  for  the  evaluation  of  the  storm  motion  product  were  taken 
from: 


Memphis  6/9/94  1611-1741  UTC 
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FIGURE  B-4.  DATA  PLOT.  JUNE  9,  1994,  165500  UTC,  MEMPHIS,  TN. 
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Shows  evaluation  data  for  storm  motion  and  extrapolated  position  products. 


B.4  Storm  Extrapolated  Position 

The  plots  used  to  evaluate  the  SEP  product  (an  example  plot  is 
shown  in  Figure  B-4)  contained  four  panels: 

1)  A  combination  of  the  current  precipitation  product  with 
SEP  leading  edges  (depicted  in  blue)  and  the  current  stora 
motion  product  (arrows  and  numeric  characters)  (upper  left  panel); 

2)  The  precipitation  product  for  ten  minutes  later  overlaid 
with  the  original  projected  leading  edges  (upper  right); 

3)  The  precipitation  product  for  15  minutes  later  overlaid 
with  the  original  projected  leading  edges  (this  panel  was 

not  used  in  the  analysis)  (lower  left); 


4)  The  precipitation  product  for  20  minutes  later  overlaid 
with  the  original  projected  leading  edges  (lower  right). 

one  plot  was  provided  for  each  update,  which  occurred  about  once 
every  two  minutes . 


For  each  plot,  the  first  panel  was  examined  to  determine  if  the 
initial  SEP  leading  edge  for  each  precipitation  cell  corresponded 
with  the  actual  leading  edge  of  the  cell.  Then  the  projected  edges 
were  checked  to  determine  consistency  with  the  storm  motion 
direction  and  speed.  Next,  the  second  panel  was  examined  to 
determine  how  well  the  SEP  ten-minute  projected  location  for  each 
cell  corresponded  with  the  actual  location  of  the  cell  leading  edge 
ten  minutes  later.  The  fourth  panel  was  examined  in  like  manner  to 
determine  how  well  the  SEP  20-minute  projected  location  for  each 
cell  corresponded  with  the  actual  location  of  the  cell  leading  edge 
20  minutes  later. 


Data  for  the  evaluation  of  the  storm  extrapolated  position  product 
were  taken  from: 


Memphis  6/9/94  1611-1741  UTC 

The  probabilities  in  Table  8  were  computed  as  follows: 

PqEI  =  Number  of  hits  /  Number  of  extrapolations 

PPFF  =  Number  of  hits  /  (Number  of  extrapolations 
^  -  Number  affected  by  evolution) 


Pea  “  Number  affected  by  evolution  / 
Number  of  extrapolations 
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pj_guj»g  B~4  shows  tho  datei  plct  foir  Juns  9  at  165500  (uppsir  Isft 
panel).  At  this  time,  there  was  a  straight  solid  blue  line  about 
80  km  long  representing  a  "leading  edge."  This  line  was  not 
following  the  level  three  edge  of  any  storm  on  the  plot.  For  this 
reason,  any  extrapolations  based  on  this  "leading  edge"  were 
considered  to  be  erroneous.  The  SEP  product  continued  from  that 
time  to  generate  erroneous  extrapolations  based  on  similar  "leading 
edges . "  These  erroneous  extrapolations  account  for  the  number  of 
misses  shown  in  Table  8.  An  algorithm  change.  Implemented  after 
the  demonstration  in  Memphis  and  for  the  demonstration  in  Orlando, 
prevented  these  erroneous  edges  from  occurring.  This  will  be 
evaluated  during  future  prototype  operations. 

Figure  B-4  also  shows  examples  of  storm  cell  growth,  decay,  and 
merging.  The  small  level  three  cell  located  at  coordinates  (330 
degrees,  47  km)  at  165500  decayed  to  level  two  by  170508  (upper 
right  panel  in  Figure  B-4)  and  to  level  one  by  171450  (lower  right 
panel  in  Figure  B-4). 

Between  20  and  60  km  south  of  the  airport  there  was  an  example  of 
celX  growth.  The  solid  blue  line  for  this  cell  (with  two  level  5 
cores)  at  165500  (Figure  B-4)  followed  the  leading  edge  (level 
three)  very  closely.  As  time  progressed,  the  edge  of  the  level 
three  contour  expanded  beyond  the  extrapolated  positions.  At 
171450  (lower  right  panel),  almost  20  minutes  later,  the  actual 
leading  edge  was  2-15  km  beyond  the  20  minute  extrapolation. 

At  165500  (Figure  B-4),  there  was  an  example  of  cell  merging. 
Consider  the  storm  cell  at  (200,52).  By  171450  (lower  right 
panel),  this  cell  had  merged  with  the  large  cell  to  its  northeast. 

B.5  Mlcroburst/Windshear  Detection 

To  aid  in  the  evaluation  of  this  product,  MIT/LL  provided  color 
maps  of  TDWR  base  velocity  at  0.1  degrees  elevation;  shear  at  0.1 
degrees  elevation  calculated  from  the  TDWR  base  velocity;  TDWR  base 
reflectivity  adjusted  to  the  1000  meter  layer;  and  vertically 
integrated  liquid  (VIL)  derived  from  TDWR  and  MCARS  base  data,  also 
in  the  1000  meter  layer.  An  example  plot  is  depicted  in  Figure  B- 
5. 

The  TDWR  performed  sector  scans  that  covered  the  airport  (e.g.,  315 
to  45  degrees)  at  a  typical  elevation  of  0.1  degrees.  The 
microburst  product  was  applied  over  this  sector.  The  detection 
range  for  the  product  was  set  at  35  km.  The  maps  provided  by 
MIT/LL  covered  only  this  sector  and  range. 

Detected  and  predicted  windshear  shapes  were  superimposed  in  red 
and  white,  respectively,  on  the  base  velocity  and  shear  plots. 

These  maps  were  updated  approximately  once  each  minute,  the  same 
update  rate  as  the  TDWR. 
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FIGURE  B-6.  DATA  PLOT,  JULY  26.  1994,  193334  UTC,  ORLANDO,  FL. 
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Shows  evaluation  data  for  microburst  detection  and  microburst  prediction  products.  Red  shapes  are  detections  and  white  shapes  are  predictions 


These  data  plots  were  analyzed  as  follows:  The  shear  map  was 
examined  for  areas  of  strong  shear  (>4  X  10"^  sec‘^).  Then  the 
velocity  map  was  examined  in  and  near  the  strong  shear  regions  for 
radial  velocity  differences  of  7.5  m/sec  (15  kt)  or  greater  (up  to 
15  m/sec  (30  kt))  for  minimal  wlndshears,  and  15  m/sec  (30  kt)  or 
greater  for  microbursts . 

Qualified  regions  were  subjected  to  a  test  for  adequate  VIL.  The 
VIL  map  was  examined  for  values  in  excess  of  three  kg/m  for 
Memphis  and  five  kg/m^  for  Orlando  within  1.5  km  of  the  region 
perimeter.  Only  those  prospective  velocity/shear  features  that 
could  be  associated  with  these  VIL  values  were  identified  as  truth. 
These  features  constituted  the  truth  set  used  in  evaluating  the 
performance  of  the  microburst  detection  and  prediction  products. 


The  following  data  were  used  in  the  evaluation  of  the 
microburst/wlndshear  detection  product: 
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6/3/94 

6/22/94 

7/4/94 

7/28/94 

8/9/94 

8/15/94 


2110-2220  UTC 

2000-2045 

1840-1940 

1835-1915 

1945-2120 

2010-2040 


B.6  Microburst  Prediction 

The  plots  used  to  evaluate  the  microburst/wlndshear  detection 
product  were  also  used  in  evaluating  the  microburst  prediction 
product.  Figure  B-5  contains  an  example  plot. 

Events  verified  in  the  microburst  detection  product  analysis  were 
used  as  truth  for  verifying  the  prediction  product.  (There  were  no 
missed  detections  for  the  examined  data  set,  see  sections  5. 2. 6. 7 
and  B . 5 ) . 


MIT/LL  product  literature  indicated  that  the  prediction  product 
does  not  predict  events  on  the  edges  of  data  (within  three  degrees 
in  azimuth  or  five  kilometers  in  range) .  The  range  for  predictable 
events  was  5-30  km,  whereas  the  range  for  detectable  events  was  0- 
35  km.  Events  outside  the  predictable  event  area  were  removed  from 
the  truth  set. 


For  each  microburst,  plots  as  early  as  nine  minutes  prior  to  a 
microburst  occurrence  were  checked  for  predictions.  If  one  or  more 
predictions  were  found,  then  the  microburst  was  considered  to  have 
been  predicted.  The  prediction  must  have  overlapped  or  been  within 
one  kilometer  of  the  microburst. 


B.7  Gust  Front  Detection 
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Plots  used  in  the  evaluation  depicted  two  panels; 

1)  TDWR  base  velocity  at  0.3  degree  tilts  to  a  range  of  60 

km; 

and 


2)  TDWR  base  reflectivity. 

These  plots  were  provided  at  five  minute  intervals.  An  example 
Plot  is  shown  in  Figure  B-6.  On  the  plots,  detected  gust  frqnts 
are  represented  by  white  lines  superimposed  on  the  base  data. 

Data  analysis  for  the  gust  front  product  proceeded  as  follows; 

Each  plot  was  examined  for  gust  fronts.  This  involved  searching 
for  "thin  lines"  in  the  reflectivity  data  and  convergence  lines  in 
the  velocity  data.  Each  analyst  determined  truth  gust  front  was 
outlined  in  red  on  both  panels. 

Probabilities  of  detection  (Pdl)  false  alarm  (Pfal)/  based  on 

length,  were  computed  as  follows; 

=  sum  [lengths  of  detected  front  segments  determined  to 
be  hits]  / 

sum  (lengths  of  truth  fronts] 

PpAL  =  sum  [lengths  of  detections  determined  to  be  false]  / 
(sum  [lengths  of  detected  front  segments  determined  to 

be  hits]  + 

sum  [lengths  of  detections  determined  to  be  false]) 


Summations  apply  to  the  entire  data  set. 

Data  used  in  evaluating  the  gust  front  detection  product  were; 
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If 

II 

It 

It 

II 


7/26/94  1826-1933  UTC 

7/27/94  1941-2035 

8/9/94  1948-2101 

8/11/94  1731-1810 

8/13/94  1759-1931 

8/14/94  2017-2051 


B.8  Teinninal  Winds 


For  the  evaluation,  MIT/LL  provided  three  different  plots; 

1)  Gridded  terminal  winds  arrows  superimposed  on  a  color  coded 
NEXRAD  base  velocity  map; 
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FIGURE  B-6.  DATA  PLOT,  JULY  26,  1994,  193334  UTC,  ORLANDO,  FL. 
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Shows  evaluation  data  for  gust  front  product.  White  lines  are  product  detections.  Red  lines 
enclose  truth  gust  fronts. 


2)  Arrows  superimposed  on  a  NEXRAD  base  reflectivity  map; 


and 


3j  Arrows  superimposed  on  a  TDWR  base  velocity  map. 

The  arid  was  120  X  120  km^,  with  winds  spaced  4  km  apart 
horizontally.  Three  levels  were  provided:  1800,  ^400,  and  120^ 
feet.  Updates  were  every  15  minutes.  An  example  plot  with  NEXRAD 
data  and  superimposed  gridded  terminal  winds  arrows  is  shown  in 

Figure  B-7 . 

According  to  MIT/LL,  aircraft  winds  and  timely  MAPS  forecasts  were 
not  available.  Also,  the  actual  winds  at  each  gate  that  appeared 
on  the  GSD  were  not  provided. 

The  analysis  for  the  terminal  winds  product  involved  visual 
examination  of  the  two  velocity  plots  to  determine  the  degree  of 
correlation  between  the  gridded  winds  and  the  radar  base 
velocities.  Reflectivity  was  examined  to  determine  the  validity  of 

the  base  velocity. 


For  the  terminal  winds  product  evaluation,  the  following  data  were 
analyzed: 


Memphis 

II 

II 


6/3/94 

6/9/94 

6/17/94 


1700-1800  UTC 
1815-2000 
2100-2200 


B.9  Storm  Cell  Information 

The  Storm  Cell  Information  (SCI)  product  associates  attributes  with 
individual  storm  cells.  These  attributes  are  hail,  lightning, 
mesocyclone,  and  echo  tops.  Section  B.9.1  contains  discussion 
relative  to  product  performance  in  associating  cells  and 
attributes.  Section  B.9. 2  contain  discussions  related  to  hail 
detection  performance  analysis. 


B.9.1  Attribute-to-Cell  Association 

MIT/LL  provided  a  sequence  of  plots  (an  example  plot  is  provided  in 
Figure  B-8),  each  containing  four  panels: 

1)  AP-edited  precipitation  map  with  cell  contours  (upper  left 
panel) ; 

2)  Hail  probability  map  with  cell  contours  (upper  right); 
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FIGURE  B-7,  DATA  PLOT,  JUNE  16,  1994,  210000  UTC,  MEMPHIS,  TN. 
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NEXRAD  base  velocity  map  is  in  background. 


FIGURE  B-8.  DATA  PLOT,  JUNE  9,  1994,  172404  UTC,  MEMPHIS,  TN. 
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Shows  evaluation  daia  for  storm  cell  informanon  product.  Upper  left  panel  contains 
contoured  storm  cells  overlaid  on  AP  edited  precipitation.  Upper  right  contains  hail  probability 
map  overlaid  with  storm  cell  contours.  Lower  left  shows  the  lightning  flash  map  and  lower 
right  the  echo  top  map,  both  overlaid  with  storm  contours. 


3)  Lightning  flashes  and  mesocyclone  map  with  cell  contours 
(lower  left); 


and 

4) 


Echo  tops  map  with  cell  contours  (lower  right). 


Each  map  was  overlaid  with  contours  (outlines) 
reflectivity  in  each  significant  storm  cell, 
updated  about  once  each  minute. 


of  the  highest  level 
These  plots  were 


The  AP-edited  precipitation  map  with  overlaid  cell  contours  was 
examined  for  correspondence  between  the  contours  and  actual  cell 
boundaries.  The  other  maps  were  checked  for  correct  associations 
between  the  contoured  cells  and  attributes.  A  probability  of 
severe  hail  (POSH)  of  50  percent  or  higher  (color  coded  to  the 
right  of  the  hail  map)  was  necessary  for  hail  detection  and  hall 
attribute-to-cell  association.  The  association  algorithm  required 
that  an  ITWS  detection  be  within  the  cell  contour  in  order  for  the 
detection  to  be  associated  with  the  cell. 


Hail.  There  were  hail  product  detections,  and  these  detections 
were"associated  with  cells.  However,  some  detections  were  outside 
the  contours.  An  example  of  this  is  shown  in  Figure  B-8,  in  the 
hall  probability  map  in  the  upper  right  panel  (time:  172404).  An 
area  of  POSH  between  50  and  60  percent  is  evident  in  the  hail 
probability  map,  but  there  is  no  cell  with  which  to  associate  it. 
This  POSH  area  is  at  coordinates  (305,92),  where  the  first 
coordinate  is  the  azimuth  relative  to  the  airport  center  in 
dsgreGS ,  and  the  second  coordinate  is  the  range  from  the  airport 
center  in  kilometers. 


There  was  also  no  association  in  the  next  update,  172459,  shown  in 
Figure  B-9.  An  association  was  not  made  until  the  following 
update,  172745,  when  a  contoured  cell  appeared  and  encompassed  the 
POSH  area  (see  Figure  B-10).  There  was  no  hail  alert  for  this  POSH 
area  for  almost  three  minutes. 
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FIGURE  B-9.  SAME  AS  FIG.  B-8,  EXCEPT  172459  UTC 
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FIGURE  B-12.  SAME  AS  FIG.  B-8,  EXCEPT  172936  UTC 
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FIGURE  B-13.  SAME  AS  FIG.  B-8,  EXCEPT  173126  UTC 
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A  second  example  may  be  seen  at  172745  (see  Figure  B-10).  An  area 
Sf  50-60  percLt  POSH  was  located  at  (195,70),  at  the  south  end  of 
a  laroe  level  five  cell  with  the  hail  attribute.  In  the  next 

172840  (Figure  B-11),  the  five  cell  developed  a  small 

d^ooeS  lnLlr*of  r?o”SrLoSr?hrimarLvers!x“o?“ 

FurtLr  examination  of  Figure  B-11  indicated  ff'* 

not  move  or  decrease,  but  yet  was  no  longer  associated  with  a  cell. 

The  next  update  at  172936  (Figure  B-12), 

Without  an  association.  The  following  update  (^he  NEX^D  also 
undated  in  the  interim)  at  173126  (Figure  B-13),  showed  the  POSH 
a?ef with  a^ncJease  in  areal  extent  and  POSH  of  60-70  percent. 
However  there  was  still  no  association.  An  association  was  made 
at  173154,  when  another  contoured  cell 

POSH  area.  Again,  there,  was  no  hail  alert  for  three  minutes. 

Mesocvclone.  Mesocyclone  detections  at  175309  and  175404,  location 
T^lO^SSl  ^re  not  associated  with  any  cell.  In  fact,  there  were 
L  wntoired  cells  nearby.  Other  mesocyclone 

and  175404,  location  (225,70);  and  at  175213,  location  (205,  0) 
also  were  not  associated  with  a  contoured  cell. 

Echo  Tops.  An  example  of  a  58000  ft  echo 

associated  with  a  contoured  cell  is  shown  in  Figure  B  10.  The 
coordinates  of  the  58000  ft  echo  top  are  (252,65).  The  nearest 
contoured  cells  to  the  southeast  and  northeast  were  indicating  echo 
tops  of  only  48000  ft.  Comparing  the  ASR-9  precipitation  with  the 
echo  top  map  indicates  that  the  area  of  highest  echo  tops  is  offset 
to  the  west  from  the  contoured  area  of  highest  precipitation  level, 
other  examples  of  missed  associations  of  58000  ft  echo  top  maxima 
ocSuLerat  1?3126  at  coordinatea  (220,88)  and  (195,55)  (Figure  B- 
13).  The  nearest  contoured  cells  contained  maximum  echo  tops  of 
53000  ft.  Again,  neither  was  associated  with  a  cell. 

B.9.2  Hail  Detection 

Tfj  order  to  evaluate  the  hall  product,  ACW— 200  obtained  lists  of 
hail  detections  prepared  by  MIT/LL  from  demonstration  data.  These 
lists  orovlded  occurrence  times  and  locations  in  rectangular 
coordinates  (x,y)  and  kilometers  relative  to  the  NEXRAD.  ACW-200 
also  obtained  severe  storm  data  from  the  NWS.  These  severe  storm 
data  are  a  compilation  of  public  and  official  reports  of  severe 
weather  sightings.  Including  those  of  hail  0.75  inches  or  larger  on 
the  ground!  It  was  these  hail  sizes  that  the  NSSL  hall  product 
was,  according  to  NSSL,  designed  to  detect. 

The  NWS  reports  were  the  only  truth  available  for  verifying  the 
detection  capability  of  the  hail  product.  Thm  were,  no  doubt, 
cases  for  which  severe  hail  was  detected  by  ITWS  and  reached  the 
ground,  but  was  not  observed  or  officially  reported.  Verifying 
these  types  of  events  would  have  required  many  human  and  automated 


observers  strategically  located  in  the  ITWS  coverage  area,  and  is 
beyond  the  scope  and  resources  of  this  study. 


Analvsls  for  the  hail  product  evaluation  proceeded  as  follows; 
R2«anjul«  coSrdJnatel  {x,y)  in  kilometers  of  each  MWS  hail  event 
location  relative  to  NEXRAD  were  determined  from  US  Army  Corps  of 
Engineers  1:250,000  topographical  maps.  These  coordinates  were 
compared  with  those  of  the  product  hail  detections.  Searches  of 
from  one  hour  before  to  an  hour  after  the  NWS  event  time  were 
made  to  find  corresponding  hall  detections. 


For  the  hail  detection  product,  these  data  were  analyzed; 


Memphis 


6/5/94 

6/9/94 

6/12/94 

6/26/94 

6/28/94 


A  list  of  all  NWS  compiled  hail  siting  reports  for  the  Memphis 
demonstration  coverage  area  in  June  1994  is  contained  in  table 
Table  B-2  shows  detections  by  the  hall  product  that  correspond  to 
the  NWS  hail  siting  reports.  These  hail  product  detections  were 
taken  from  a  list  of  all  hail  product  detections  compiled  by  MIT/LL 
for  the  hall  product. 
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TABLE  B-1.  National  Weather  Service  Hail  Sightings  in  Memphi 
ITWS  Coverage  Area  During  the  Demonstration  (June  1994). 


Event 

# 

Date 

Location 

Time 

(UTC) 

Hail 

Size 

1 

6/5 

Horseshoe  Lake, 

AR  1945 

1.75 

2 

6/5 

Walcott,  AR 

2300 

0.75 

3 

6/9 

Savage,  MS 

1658 

1.00 

4 

6/9 

Batesvllle,  MS 

1720 

1.00 

5 

6/9 

Covington,  TN 

1820 

0.75 

6 

6/9 

Memphis,  TN 

1840 

1.00 

7 

6/9 

Belen,  MS 

1815 

0.75 

8 

6/9 

Potts  Camp,  MS 

1830 

0.75 

9 

6/12 

Caldwell,  AR 

2230 

0.75 

10 

6/26 

Cash,  AR 

1010 

1.00 

11 

6/26 

Harrisburg,  AR 

1100 

0 . 88 

12 

6/26 

Wynne,  AR 

1147 

1.00 

13 

6/26 

Parkin,  AR 

1153 

1.75 

14 

6/26 

Forrest  City,  AR  1201 

0.75 

15 

6/26 

Brighton,  TN 

1301 

0.75 

16 

6/26 

Bradford,  TN 

2230 

1.75 

17 

6/26 

Nutbush,  TN 

2303 

0.75 

18 

6/26 

Brownsville,  TN 

2355 

1.75 

19 

6/28 

Monet te,  AR 

1305 

0.75 

20 

6/28 

Osceola,  AR 

1327 

0.75 

21 

6/28 

Wilson,  AR 

1358 

0.88 

22 

6/28 

Memphis,  TN 

1425 

1.00 

TABLE  B-2.  Product  Detected  Hall  Events  Corresponding  to 
NWS  Sightings  in  Memphis  ITWS  Coverage  Area  (June  1994) 


NWS  NWS  Event  Product  Event  Time 

Event  Location  Location  (UTC) 

#  (x,y)  km  (x,Y)  km 

from  NEXRAD  from  NEXRAD  _ 


1 

(-42,-50) 

2 

(-78,78) 

3 

(-33,-87) 

4 

(-10,-118) 

5 

(20,21) 

6 

(Memphis) 

7 

(-47,-123) 

8 

(50,-85) 

9 

(-91,-35) 

10 

(-102,48) 

11 

(-83,20) 

12 

(-90,-17) 

13 

(-67,-12) 

14 

(-88,-40) 

15 

(12,13) 

16 

(100,82) 

17 

(43,38) 

18 

(56,26) 

19 

(-47,60) 

20 

(-13,38) 

21 

(-25,23) 

22 

(Memphis) 

(-46,-49)  1859 

(-76,82)  2303 

(-31,-78)  1654 

(-13,-114)  1812 

No  corresponding  detection 
No  corresponding  detection 
(-40,-117)  1753 

(46,-95)  1753 

(-80,-33)  2214 

(-103,49)  1018 

(-88,18)  1108 

(-87,-17)  1143 

(-79,-14)  1138 

(-76,-36)  1158 

(6,25)  1238 

(90,80)  2222 

(37,31)  2252 

(49,21)  2326 

ITWS  not  operational 
ITWS  not  operational 
ITWS  not  operational 
(1,-29)  1424 


B.IO  Tornado 


MIT/LL  provided  a  list  of  product  detected  tornadoes 
along  with  event  dates ^  times /  and  locations.  They  also 
provided  another  list  with  analyst  determined  truth 
events.  These  truth  events  were  determined  by  examining 
base  TDWR  and  NEXRAD  velocity  data  for  tornado 
signatures. 

The  tornado  data  were  analyzed  by  reviewing  MIT/LL 's 
analysis/  determining  the  number  of  hitS/  misses /  and 
false  detections/  and  computing  the  Pq  and  PpA 
data  set . 

The  .tornado  product  was  evaluated  using  these  data. 

Memphis  6/8/94 

••  6/9/94 

••  6/26/94 

••  6/27/94 

B.ll  Terminal  Weather  Text  Message 

For  the  evaluation  of  this  product/  MIT/LL  provided  color 
plots  showing  the  ITWS  GSD  display  out  to  16  nm  in  range/ 
along  with  the  ITWS  graphics  message/  RDT  display, 
terminal  text  message,  and  corresponding  NWS  observation. 
The  plots  were  updated  every  five  minutes.  Ah  example 
plot  is  depicted  in  Figure  B-14. 

For  each  plot,  the  information  in  the  terminal  text 
message  was  compared  with  the  graphical  information  on 
the  GSD  for  consistency. 

The  terminal  weather  text  message  product  was  evaluated 
using  the  following  data: 

Memphis  6/7/94  0308-0625  UTC 

This  appendix  contains  details  of  the  analyses  and 
analysis  methods  used  to  obtain  the  results  presented  in 
the  meteorological  performance  evaluation  section, 
section  5.2.6. 
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"  ,  tyj . . . . . 


FIGURE  B-14.  DATA  PLOT,  JUNE  7,  1994,  0308  UTC,  MEMPHIS,  TN. 

B-27 


Shows  evaluation  data  for  terminal  weather  text  message  product. 


APPENDIX  C 


INTEGRATED  TERMINAL  WEATHER  SYSTEM 
OPERATION  MEMPHIS  DEMVAL  1994 


Integrated  Terminal  Weather  System  Operation 
Memphis  DEMVAL  1994 


DATE 

ITWS 

OPERATION 

TDMR 

OPERATION 

NEXRAD 

OPERATION 

ASR-9 

OPERATION 

5/23 

1700  -  0000 

DOWN 

1700  -  0000 

1700  -  0000 

5/24 

1710  -  0011 

DOWN 

1710  -  0011 

1710  -  0011 

5/25 

1721  -  0132 

1721  -  0132 

1721  -  0132 

1721  -  0132 

5/26 

0857  -  1400, 

0857  -  1112, 

0857  -  1044, 

0857  -  1400, 

1700  -  0000 

DOWN 

1700  -  0000 

1700  -  0000 

5/27 

1700  -  0000 

1700  -  0000 

1700  -  0000 

1700  -  0000 

5/30 

1410  -  0338 

DOWN 

1410  -  0338 

1410  -  0338 

5/31 

1701  -  0000 

1701  -  0000 

1701  -  2105, 
2305  -  0000 

1701  -  0000 

6/1 

1701  -  0000 

1701  -  0000 

1949  -  0000 

1701  -  0000 

6/2 

1700  -  0408 

1700  -  0408 

1700  -  0408 

1700  -  0408 

6/3 

1700  -  0014 

1700  -  0014 

1700  -  0014 

1700  -  0014 

6/4 

1700  -  0003 

||BSZ!E]il^E££BH 

1700  -  0003 

1700  -  0003 

6/5 

1700  -  0115 

1700  -  0115 

1700  -  0115 

1700  -  0115 

6/6 

1700  -  1300 

1700  -  1300 

1700  -  1036 

1700  -  1300 

6/7 

1600  -  0003 

1600  -  1953 

1600  -  0003 

1600  -  0003 

6/8 

1633  -  2330 

1633  -  2330 

1633  -  2330 

1633  -  2330 

6/9 

1225  -  0000 

1225  -  0000 

1225  -  0000 

1225  -  0000 

6/10 

1700  -  0000 

1700  -  0000 

1700  -  0000 

1700  -  0000 

6/12 

2225  >  0200 

2225  -  0200 

2225  -  0200 

2225  -  0200 

6/13 

1600  -  0002 

1600  -  0002 

1730  -  0002 

1600  -  0002 

6/14 

1705  -  0005 

1705  -  2300 

1705  -  0005 

1705  -  0005 

6/15 

1658  -  0001 

1658  -  0001 

1658  -  0001 

1658  -  0001 

6/16 

1658  -  0142 

1658  -  0142* 

1658  -  0142 

1658  -  0142 

6/17 

1700  -  0125 

1700  -  0125 

1700  -  0125 

1700  -  0125 

6/18 

1823  -  0130 

1823  -  0130 

1823  -  0130 

1823  -  0130 

6/19 

1615  -  0100 

1615  -  0100 

1615  -  0100 

1615  -  0100 

6/20 

1700  -  0007 

1700  -  0007 

1700  -  2152, 
2220  -  0007 

1700  -  0007 

6/21 

1700  -  0102 

1700  -  0102 

1700  -  0102 

1700  -  0102 

6/22 

1700  -  0001 

1700  -  0001 

1700  -  0001 

1700  -  0001 

6/23 

1630  -  0003 

1630  -  0003 

1630  -  2110, 
2137  -  0003 

1630  -  0003 

6/24 

0654  -  1315, 

0654  -  1315, 

0654  -  1131, 
1200  -  1315, 

0654  -  1315, 

1700  -  2300 

1700  -  2300 

1700  -  2300 

1700  -  2300 

6/26 

1109  -  0705 

1109  -  0705 

1109  -  1642, 
1800  -  0705 

1109  -  0705 

6/27 

1708  -  0056 

1708  -  0056 

1708  -  0056 

1708  -  0056 

6/28 

1335  -  0420 

1335  -  0420^ 

1335  -  0420 

1335  -  0420 

6/29 

1058  -  0030 

1058  -  1830, 
1954  -  0030 

1058  -  0030 

1058  -  0030 

6/3on 

1531  -  0326 

1531  -  0326 

1531  -  0326 

1531  -  0326 

7/1 

1700  -  0004 

1700  -  0004 

1700  -  0004 

1700  ~  0004 

7/2 

0045  -  1630, 
2330  -  0539 

0045  -  1630, 
2358  -  0539 

0045  -  1630, 
2330  -  0539 

0045  -  1630, 
2330  -  0539 

7/3 

1246  -  0346 

1246  -  0346  n 

1246  -  0346 

1246  -  0346 

7/4 

1700  -  0000 

1700  -  0000 

1700  -  0000 

1700  -  0000 

7/5 

m^  ff 

1645  -  0253 

7/6 

1708  -  0034 

1708  -  1942, 
2026  -  0034 

1708  -  0034 

1708  -  0034 

7/7 

1700  -  0050 

1700  -  0050 

1700  -  0050 

1700  -  0050 

7/8 

1225  -  0014 

1225  -  0014 

1225  -  2014, 
2025  -  0014 

1225  -  0014 

7/9 

1609  -  2250 

1609  -  1735, 
1808  -  2250 

1609  -  2250 

1609  -  2250 

7/11 

1638  -  0256 

1638  -  0256 

1638  -  0256 

1638  -  0256 

7/12 

1454  -  0304 

1454  -  1630 

1454  -  0304. 

1454  -  0304 

7/13 

1150  -  0000 

DOWN 

1150  -  0000 

1150  -  0000 

7/14 

1310  -  1429, 
1700  -  0500 

DOWN 

1310  -  1429, 
1700  -  0500 

1310  -  1429, 
1700  -  0500 

7/15 

1612  -  0310 

DOWN 

1612  -  0310 

1612  -  0310 

7/16 

DOWN 

1225  -  2014 

1225  -  0014 

7/17 

2140  -  0240 

DOWN 

2140  -  0240 

2140  -  0240 

7/18 

IBVZtlilHiLdllllfMi 

DOWN 

1700  -  0000 

1700  -  0000 

7/19 

1705  -  0000 

DOWN 

1836  -  0000 

1705  -  0000 

7/20 

1648  -  0000 

DOWN 

1648  -  0000 

1648  -  0000 

7/21 

1650  -  0558 

DOWN 

1650  -  0558 

1650  -  0558 

7/22 

1601  -  0025 

DOWN 

1601  -  0025 

1601  -  0025 

*  TDWR  had  intermittent  failures  throughout  the  operational 
period  due  to  a  problem  in  the  SBC  in  the  servo 
control  unit. 


-  Number  of  hours  ITWS  operational  :530hr.,  53min. 

-  Number  of  hours  TDWR  available  :378hr./  43min.(  71%  ) 

-  Number  of  hours  NEXRAD  available  :510hr.,  31mln. (  96%  ) 

-  Number  of  hours  ASR-9  available  :530hr.,  53min.(  100%  ) 

-  Dates  ITWS  was  not  operational  ;  5/28,  5/29,  6/11,  6/25, 
7/10. 


The  TDWR  was  not  available  for  thirteen  days  of  the  DEMVAL 
period.  No  signifiacant  weather  was  observed  during  six  of 
the  days.  The  dates  no  events  occur ed  are:  5/23,  5/24, 
7/13,  7/14,  7/17,  7/18,  7/19. 
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APPENDIX  D 


INTEGRATED  TERMINAL  WEATHER  SYSTEM 
OPERATION  ORLANDO  DEMVAL  1994 


Integrated  Terminal  Weather  System  Operation 
Oriando  DEMVAL  1994 


DATE 


7/11 


7/12 


7/13 


7/14 


7/15 


7/16 


7/17 


7/18 


7/19 


7/20 


7/21 


7/22 


7/23 


7/24 


7/25 


7/26 


7/27 


7/28 


7/29 


7/30 


7/31 


8/1 


8/10 


8/11 


8/12 


8/13 


8/14 


8/15 


8/16 


8/17 


8/18 


8/19 


ITWS 

OPERATION 


1600  -  2301 


1559  -  0157 


1601  -  0218 


1600 


1723 


2353 


2310 


1506 


1553 


1538 


1557 


1733 


1645 


1552 


1513 


1559 


1556 


1602 


1530 


1600 


2318 


0011 


2300 


2305 


2339 


2108 


2300 


0001 


2307 


2300 


0122 


2324 


2300 


1523  -  2305 


1559 


1640 


1600 


1556 


1558 


2300 


2145 


2342 


2300 


2300 


1555  -  0031 


1522 


1441 


1618 


1732 


1401 


1553 


2301 


2119 


2218 


2303 


2305 


0009 


1554  -  0152 


TDWR 

NEXRAD 

ASR-9 

OPERATION 

OPERATION 

OPERATION 

1600  -  2301 


1601  -  0218 


1600  -  2301 


1601 

1912 


1840 

0218 


DOWN  1 

2014 

- 

2310 

1741 

- 

2222 

1557 

- 

2241 

1506 

- 

2318 

155'3 

- 

0011 

1538 

- 

2300 

1557 

- 

2305 

1733 

2339 

1645 

- 

2008 

1559 

- 

2302 

1552 

- 

2300 

1513 

- 

0001 

1559 

2307 

1556 

- 

2300 

1602 

- 

0122 

1530 

2324 

1600 

2300 

1555 

- 

2340 

1559 

- 

2300 

1640 

■ 

2145 

1600 

- 

2342 

1556 

- 

2300 

1558 

- 

2217 

2234 

- 

2300 

1555 

- 

0031 

1522 

- 

2301 

1441 

- 

2119 

1618 

- 

2218 

1732 

- 

2303 

1401 

- 

2305 

1553 

0009 

1557 

- 

0021 

1554 

- 

0152 

1506 


1553 


1538 


1557 


-  22 


-  2318 


-  0011 


-  2300 


-  2305 


1552 


1513 


1559 


1602 


1530 


-2028 


-  0001 


-  2004 


-  0100 


-  2324 


1600 


1556 


1558 


2300 


1818 

2145 


2222 


2300 


2300 


1555  -  0031 


1522 


1441 


1618 


1732 


1401 


1553 


2301 


2119 


2218 


2303 


2305 


0009 


1554  -  0152 


1600  -  2301 


1559  -  0157 


1601  -  0218 


1600  -  2353 

1600  -  2353 

1723  -  2310 

1723  -  2310 

1557 

- 

2241 

1506 

- 

2318 

1553 

- 

OOli 

1538 

- 

2300 

1557 

- 

2305 

1733 

2339 

1645 

- 

2108 

1559 

- 

2302 

1552 

- 

2300 

1513 

- 

0001 

1559 

- 

2307 

1556 

2300 

1602 

- 

0122 

1530 

- 

2324 

1600 

2300 

1523 

- 

2305 

1555 

- 

2340 

1556  -  2354 

1559 

— 

2300 

1640 

2145 

1600 

- 

2342 

1556 

- 

2300 

1558 

2300 

1555 

- 

0031 

1  1559  -  2232 

1522 

— 

2301 

1441 

- 

2119 

1618 

- 

2218 

1732 

- 

2303 

1401 

- 

2305 

1553 

0009 

1557 

- 

0021 

1554 

- 

0152 

Number 

Number 

Number 

Number 


of  hours  ITWS  operational  :  297  hr.,  07  min. 
of  hours  TDWR  available  :  285  hr.,  11  min. 
of  hours  NEXRAD  available  :  279  hr.,  43  min. 
of  hours  ASR“9  available  :  297  hr.,  07  min. 


(96  %) 
(94  %) 
(100  %) 


